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STATEMENT FROM THE U.S. DEPARTMENT OF EDUCATION 



Data were collected from the first year of the SES pilot (2005-06) in Chicago and Boston, but they are 
not included in this evaluation report or issued as part of a separate evaluation report because there were 
some issues with and some gaps in the data collected. Thus, this evaluation data was used for internal 
purposes only. 

Subsequentiy, the Department took steps to improve data collection with participating districts. 
Specifically, the Department (a) clarified the data requirements with the participating districts; (b) 
instituted a process to contact the districts earUer in the school year for the background data; and (c) 
collected only the current year achievement data later in the year. Thus, the data collection process with 
the second group of waiver grantees that is described in this report was improved. A future evaluation 
and evaluation report will examine the implementation of supplemental educational services in all five 
waiver districts (Anchorage, Boston, Charlotte-Mecklenburg, Chicago, and Hillsborough) and will review 
aU relevant data, but will focus on results from the 2006-07 school year through the 2008-09 school year. 
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EXECUTIVE SUMMARY 



Under the Hkmentary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Eeft Behind 
Act of 2001 (NCEB), Tide I schools that do not make adequate yearly progress (AYP) for two 
consecutive years are identified for school improvement. If an identified Title I school does not make 
AYP while in that status and thus enters the second year of school improvement status, its district must 
offer students from low-income families in the school the opportunity to receive free supplemental 
educational services, such as tutoring, remediation or other academic enrichment that are provided in 
addition to instruction provided during the school day. The district must also offer supplemental 
educational services to students from low-income families in schools that continue not to make AYP and 
are identified for corrective action or restructuring. Parents of eligible students may choose a provider 
from the state-approved list and the school district pays for the services. Many types of organizations 
are eligible to apply to become state-approved supplemental educational service providers, including 
for-profit agencies, not-for-profit groups, faith-based and community-based organizations, private 
schools, and public schools and districts. 

Under federal regulations, school districts that have been identified for improvement or corrective action 
are not eligible to be supplemental educational service providers. However, the U.S. Department of 
Education has granted waivers to five such districts, on a pilot basis, to allow those districts to provide 
supplemental educational services. Boston, Massachusetts and Chicago, Illinois received waivers 
beginning in the 2005—06 school year; Hillsborough County, Florida and Anchorage, Alaska received 
waivers beginning in the 2006—07 school year; and Charlotte-Mecklenburg, North Carolina received a 
waiver beginning in the 2008-09 school year. These pilot districts are eligible to serve as providers in 
exchange for expanding students’ access to supplemental educational services and providing student 
achievement data for an evaluation to examine program effectiveness. The goals of this pilot program 
are to help ensure that more eligible students receive supplemental educational service, and to provide 
more accurate and comprehensive information to the U.S. Department of Education on the 
effectiveness of supplemental educational services in improving student academic achievement. 

This report examines the implementation of supplemental educational services in two of the waiver 
districts. Anchorage and Hillsborough, in 2006—07, including issues relating to student participation, 
achievement gains, and communication with parents. A future report will examine the implementation 
of supplemental educational services in all five waiver districts (Anchorage, Boston, Charlotte- 
Mecklenburg, Chicago, and Hillsborough) through the 2008—09 school year. 

The central evaluation question for this report is whether students served by the Anchorage and 
Hillsborough County district providers show achievement gains that are at least comparable to those of 
students served by non-district providers. This report also examines the effects of supplemental 
educational services more generally by evaluating whether, and to what extent, supplemental educational 
services participants in these two districts show achievement gains relative to students who do not 
participate. Finally, the report examines whether student achievement gains vary by individual provider 
or by student characteristics, such as race/ ethnicity, limited English proficiency, and special education 
status. 

With respect to the analysis of student achievement, the analyses presented here apply to a subset of 
students in each district — those in tested grades only. In Anchorage, students in grades 3—9 took the 
Alaska Standards Based Assessment (SB A) in 2004—05. In subsequent years, 10th graders were tested as 
well. Prior to 2004—05 only students in grades 3, 6, and 8 participated in standardized tests, and the test 
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administered was the Alaska Benchmark Exam^, not the SBA. In Hillsborough, students in grades 1—12 
participated in Florida’s Comprehensive Assessment Test (FCAT) from 2002—03 through 2004—05. In 
2005—06 and 2006—07 this test was administered to students in grades 2—12 and 3—12, respectively. 

Given these data, the achievement analyses presented in this report cover 71 and 84 percent of all 
students receiving supplemental educational services in Anchorage and Hillsborough, respectively. 

Key Findings 

Eligibility for and Participation in Supplemental Educational Services 

• In Anchorage, 6 percent of aU students in the district were eligible for supplemental educational 
services in 2006—07. In Hillsborough about 20 percent of all students in the district were eligible 
for these services in 2006—07. 

• In both districts, the percentage of eligible students participating in supplemental educational 
services increased between 2004—05 and 2006—07, reaching 1 1 percent (from 4 percent) in 
Anchorage and 14 percent (from 2 percent) in Hillsborough in 2006—07. 

• The Anchorage school district served 138 students through its own supplemental educational 
services program (45 percent of students served) in 2006—07. The district did not offer 
supplemental educational services during the previous school years (see Exhibit S.l). 

• The Hillsborough school district served 97 1 students through its own supplemental educational 
services program (16 percent of students served) in 2006—07. The district did not offer 
supplemental educational services during the previous school years (see Exhibit S.2). 

• Participants in supplemental educational services received an average of 28 and 22 hours of 
tutoring in Anchorage and Hillsborough during 2006—07, respectively. 

• In both Anchorage and Hillsborough, students who were eligible for supplemental educational 
services had significandy lower levels of prior achievement in mathematics and reading than 
students who were not eligible. 

• In both Anchorage and Hillsborough, students who participated in supplemental educational 
services in 2006—07 had significandy lower levels of prior achievement (in 2005—06) than eligible 
students who chose not to pardcipate. 

• In Hillsborough, pardcipants served by the school district in 2006—07 had higher prior 
achievement levels in 2005—06 than students served by non-district providers. In Anchorage, 
there was no clear pattern of prior achievement levels in 2005—06 between students served by 
the district and those served by non-district providers in 2006—07. 



1 The Benchmark and SBA tests have similar scale scores and proficiency markers. As explained later, scale scores were 
standardized within grade. 
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Exhibit S.1 

Number and Percentage of Supplemental 
Educational Services Participants in 
Grades K-1 2 Who Were Served 
by Type of Provider, in Anchorage, 
2004-05 Through 2006-07 




2004-05 2005-06 2006-07 



^ District Provider Non-District Provider 



Exhibit reads: In Anchorage in 
2006—07, 138 participants, or 45 percent of all 
participants, received supplemental educational 
services from the Anchorage Public Schools; 

170 students were served by non-district providers. 

Source: Anchorage School District Administrative Data, 
2002—03 through 2006—07. 



Exhibit S.2 

Number and Percentage of Supplemental 
Educational Services Participants in 
Grades K-1 2 Who Were Served by Type of 
Provider, in Hillsborough, 

2004-05 Through 2006-07 



7.000 

6.000 

5.000 

4.000 

3.000 

2.000 
1.000 

0 



6.172 




2004-05 



2005-06 



971 (16%) 



2006-07 



^ District Provider Non-District Provider 



Exhibit reads: In Hillsborough in 
2006-07, 971 participants, or 16 percent of all 
participants, received supplemental educational 
services from the Hillsborough school district; 5,201 
students were served by non-district providers. 

Source: Hillsborough School District Administrative 
Data, 2002—03 through 2006—07. 



Student Achievement Gains 

• In Anchorage, there was not a statistically significant difference in the academic achievement 
gains between participants and eligible non-participants (see Exhibit S.3). However, the 
participant group was extremely small which makes it difficult to detect significant effects. 

• In Hillsborough, students receiving supplemental educational services experienced larger 
academic gains in mathematics — but not in reading — than eligible non-participants. Participants 
improved their academic achievement by 0.05 standard deviations more than eligible 
non-participants in mathematics. 

• In both districts, students served by non-district providers showed, on average, larger gains than 
eligible non-participants in mathematics (but not in reading). Students served by the two district 
providers (Anchorage and Hillsborough) did not show achievement gains in either subject that 
were statistically distinguishable from those of eligible non-participants. 
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Exhibit S.3 

Relative Achievement Gains for Supplemental Educational Services Participants, by 
Type of Provider, in Anchorage and Hillsborough, 2002-03 Through 2006-07 





Anchorage 


Hiilsborough 




Mathematics 


Reading 


Mathematics 


Reading 



Overali Effect 


-0.05 


0.01 


0.05* 


-0.001 


District Provider 


-0.17 


-0.11 


0.05 


-0.02 


Non-District Provider 


0.18* 


0.12 


0.08* 


-0.01 



Exhibit reads: In Anchorage, students who participated in supplemental educational services in 
mathematics showed no significant differences in mathematics achievement gains from those of eligible non- 
participants. 

Notes: Scores reported are rank-based z-scores. * indicates gains that were statistically different form the gains of 
eligible non-participants at the .05 level. 

Source: Administrative data provided by the Anchorage and Hillsborough school districts, 2002—03 through 2006—07. 



• Looking at individual non-district providers, only one such provider in Hillsborough showed a 
larger achievement gain (in mathematics) relative to the gains of eligible non-participants. No 
individual provider in Anchorage showed any significant gains relative to eligible non- 
participants. 

Overall, there was not a consistent relationship between the amount of services received and 
achievement gains in either district. 

• In both districts, most student subgroups did not show significant achievement gains. 

District Communication with Parents 

• Despite the use of phrases such as “free tutoring,” both Anchorage and Hillsborough sent 
families notification materials that were difficult to understand due to use of jargon (such as 
quoting the law) and the high grade level at which the documents were written. 

• Both Anchorage and Hillsborough translated some materials into languages other than English. 

• Both Anchorage and Hillsborough provided unbiased materials describing aU supplemental 
educational service providers. 
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I. INTRODUCTION 



Under the Hlementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Eeft Behind 
Act of 2001 (NCEB), Tide I schools that do not make adequate yearly progress (AYP) for two 
consecutive years are identified for school improvement. If an identified Title I school does not make 
AYP while in that status, and thus enters the second year of school improvement status, its district must 
offer students from low-income families in the school the opportunity to receive free supplemental 
educational services, such as tutoring, remediation or other academic instruction provided outside the 
regular school day. The district must also offer supplemental educational services to students from low- 
income families in schools that continue not to make AYP and are identified for corrective action or 
restructuring. Parents of eligible students may choose a provider from the state-approved list and the 
school district pays for the services. Many types of organizations are eligible to apply to become 
state-approved supplemental service providers, including for-profit agencies, not for-profit groups, 
faith-based and community-based organizations, private schools, and public schools and districts. Under 
federal regulations, school districts that have been identified for improvement or corrective action are 
not eligible to be supplemental educational service providers. However, the U.S. Department of 
Education has granted waivers to five such districts, on a pilot basis, to allow those districts to provide 
supplemental educational services. Boston, Massachusetts and Chicago, Illinois received waivers 
beginning in the 2005—06 school year; Hillsborough County, Florida and Anchorage, Alaska received 
waivers beginning in the 2006—07 school year; and Charlotte-Mecklenburg, North Carolina received a 
waiver beginning in the 2008—09 school year. 

This report examines the implementation of supplemental educational services in two of the waiver 
districts. Anchorage and Hillsborough, in 2006—07, including issues relating to student participation, 
achievement gains, and communication with parents. A future report will examine the implementation 
of supplemental educational services in aU five waiver districts (Anchorage, Boston, Charlotte- 
Mecklenburg, Chicago, and Hillsborough) through the 2008—09 school year. 

The central evaluation question for this report is whether students served by the Anchorage and 
Hillsborough County district providers show achievement gains that are at least comparable to those of 
students served by non-district providers. This report also examines the effects of supplemental 
educational services more generally by evaluating whether, and to what extent, supplemental educational 
services participants in these two districts show achievement gains relative to students who do not 
participate. Finally, the report examines whether achievement gains vary by individual provider, amount 
of services received, or by student characteristics, such as race/ ethnicity, limited English proficiency, and 
special education status. 

Evaluation Questions 

These analyses rely on student-level achievement data from the 2002—03 through 2006—07 school years 
plus student-level data on participation in supplemental educational services in 2004—05 through 
2006-07 to address the following sets of evaluation questions: 

Eligibility for and Participation in Supplemental Educational Services 

1 . Did the percent of eligible students participating in supplemental educational services during 
the pilot year increase from the prior year(s)? 
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2. How were participating students distributed among providers? How do students served by 
the districts’ supplemental educational services programs compare to students served by 
non-district providers? 

3. Were there significant demographic or academic differences between students served by 
different providers, specifically those served by the districts’ programs compared to those 
served by non-district providers? 

Student Achievement 

1 . Are gains in achievement for students receiving supplemental educational services from 
district providers comparable to gains for students served by non-district providers? 

2. How do levels of achievement vary based on student eligibility for, and participation in, 
supplemental educational services? 

3. Controlling for amount of exposure to supplemental educational services, are reading and 
mathematics gains for students served by district providers comparable to students served by 
non-district providers? 

District Communication with Parents 

1 . Are districts communicating with parents about the availability of supplemental educational 
services in a manner that is clear, user friendly, and accessible? 

2. Is the nature of district communication unbiased with respect to particular providers, 
specifically non-district providers versus the district-provided services? 

Overview of the Supplemental Educational Services Provisions of NCLB 

A key objective of Tide I of the ESEA, as amended by NCEB, is to provide greater parental choice to 
families with children attending Tide I schools idendfied for improvement, correcdve acdon, or 
restructuring. One opdon is for parents to enroll children in supplemental educadonal services, such as 
tutoring, remediadon, or other supplemental academic enrichment that are in addidon to instrucdon 
provided during the school day. Students from low-income families are eligible for supplemental 
educadonal services if they attend a Tide I school that is in the second year of school improvement, in 
correcdve acdon, or in restructuring. Parents of eligible students may choose a provider from the 
state-approved Ust, which may include for-profit agencies, not-for-profit groups, faith-based and 
community-based organi 2 adons, private schools, and public schools and districts. Districts that are 
required to offer supplemental educadonal services must pay for these services and spend, subject to 
demand, an amount equal to at least 20 percent of their Tide I, Part A aUocadon on supplemental 
educadon services and transportadon for public school choice. 2 School districts must consult with 
providers and parents to establish achievement goals for students, and providers are required to measure 
progress towards these goals and communicate with parents regarding their child’s progress. States are 
responsible for monitoring and evaluadng provider performance, and may use monitoring and evaluadon 
results to remove from the state-approved list a provider that fails, for two consecudve years, to 
contribute to increasing students’ academic proficiency. 



2 In Title I schools that have been identified for school improvement, corrective action, or restructuring, the district 
must offer students the option to transfer to another public school in the district that has not been so identified. AH 
students in identified schools are eligible for this option, and the district must provide transportation for participating 
students, subject to available funding. 
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Few studies to date have examined the effect of supplemental educational services on student 
achievement. One study by the U.S. Department of Education found that, on average, across seven 
large, urban districts, participation in supplemental educational services had a statistically significant 
effect on students’ achievement in mathematics (a gain of 0.09 standard deviations) and reading (a gain 
of 0.08 standard deviations) relative to eligible students who did not participate.-^ Gains by students 
served by district versus non-district providers varied, but showed no clear pattern. 

Under federal regulations, school districts that have been identified for improvement or corrective action 
are not eligible to be supplemental educational service providers. However, the U.S. Department of 
Education granted waivers to five such districts, on a pilot basis, to allow those districts to provide 
supplemental educational services. These pilot districts are: Boston, Massachusetts; Chicago, Illinois; 
Hillsborough County, Florida; Anchorage, Alaska, and Charlotte-Mecklenburg, North Carolina. These 
pilot districts are eligible to serve as providers in exchange for expanding students’ ability to access 
supplemental educational services and providing student achievement data for an evaluation to examine 
program effectiveness. As part of the waiver, districts agreed to provide: 

• Early notification to parents of their children’s eligibility to participate in supplemental 
educational services; 

• Extended enrollment periods so that parents can make the best choice for their child; 

• The use of district facilities by non-district providers for a reasonable fee; and 

• Academic data to an independent third party for evaluation of the effectiveness of supplemental 
educational services. 

The goals of the pilot are to help ensure that more eligible students are receiving supplemental 
educational services, and to provide more accurate and comprehensive information on the effectiveness 
of supplemental educational services provided by districts in improvement in improving student 
academic achievement. 

This report on the pilot waivers examines the implementation of supplemental educational services in 
the Anchorage and Hillsborough County school districts, including issues relating to student 
participation, achievement gains, and communication with parents. 

Data Sources 

Analyses of supplemental educational services pose several challenges, including: obtaining student-level 
achievement data that are longitudinally linked and enable examination of student achievement gains 
over time, acquiring achievement data for students prior to their receiving supplemental educational 
services, and determining an appropriate comparison group for participants. Further, it may be difficult 
for districts to provide complete and accurate data on student eligibility for supplemental educational 
services. A student is eligible for supplemental educational services if the student attends a Tide I school 
that is in the second year of school improvement, in corrective action, or in restructuring, and if the 
student comes from a low-income family. To determine whether a student is from a low-income family, 
the district must use the same poverty measure that it uses to rank its Tide I schools. In most districts, 
student eligibility for free or reduced-price lunches is used to determine family income levels. However, 



3 Zimmer, R., Gill, B., Razquin, P., Booker, K., & Lockwood, J.R., (2007). State and local implementation of the No Child Lpft 
Behind Act: Volume 1-Title I school choice, supplemental educational senices, and student achievement. Washington, D.C.: U.S. 
Department of Education. 
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district datafiles on student eligibility for free or reduced-price lunch are frequentiy missing data for some 
students, in part because not all eligible students are enrolled in this program. . 

Anchorage and Hillsborough each provided data on student eligibility, participation, the provider from 
whom each student received services, and the number of hours of tutoring participants received in 
2004—05 through 2006—07 (see Exhibit 1). 

In Anchorage, student-level data on limited English proficiency and special education status were not 
available. Therefore, no separate analyses for these subgroups were conducted for Anchorage. 

Determining Eligibility and Participation 

For the purposes of these analyses, the research team established parameters to determine student 
eligibility for supplemental educational services. First, any student in a school not required to offer 
supplemental educational services (based on district designations of school eligibility) was considered 
ineligible. In addition, if student demographic data indicated that students were not eligible for services 
by virtue of not being eligible for free or reduced-price lunch, they were considered ineligible for the 
purpose of the analysis, even if they were, in fact, receiving services (the number of students removed 
from the analysis for this reason was very small). 



Exhibit 1 

Data Available from Anchorage and Hillsborough 




Anchorage 


Hillsborough 


Eligibility and Participation Data 


Eligibility 


2004-05, 2005-06, 2006-07 


2002-03, 2003-04, 2004-05, 
2005-06, 2006-07 


Participation 


2004-05, 2005-06, 2006-07 


2004-05, 2005-06, 2006-07 


Grades with Data Available 


K-12 


K-12 


Participation Data 


Provider Name and Type 


2004-05, 2005-06, 2006-07 


2004-05, 2005-06, 2006-07 


Hours of Services Received 


2004-05, 2005-06, 2006-07 


2004-05, 2005-06, 2006-07 


Subject of Tutoring 


2004-05, 2005-06, 2006-07 


2004-05, 2005-06, 2006-07 


Grades with Data Available 


K-12 


K-12 


Student Achievement Data 


Test (Year) 


Benchmark (2002-03 to 2003-04); 
SBA (2004-05 to 2006-07) 


FCAT (2002-03 to 2006-07) 


Tested Grades 


3,6, and 8 (2002-03 to 2003-04) 
3-9 (2004-05) 

3-1 0 (2005-06 to 2006-07) 


1-12 (2002-03 to 2004-05) 

2- 1 2 (2005-06) 

3- 1 2 (2006-07) 
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Measuring Achievement 



Each district provided longitudinal student achievement data from their state assessment. In Anchorage, 
all students in grades 3—9 in 2004—05, and grades 3—10 in 2005—06 and 2006—07 took the Alaska 
Standards Based Assessments (SBAs). Before that, achievement data was only available for students in 
grades 3, 6, and 8, on the previous state assessment, the Alaska Benchmark Exam. Hillsborough 
provided reading and mathematics scale scores on Florida’s Comprehensive Assessment Test (FCAT) 
for all students in grades 1—12 from 2002—03 through 2004—05, grades 2—12 in 2005—06, and grades 3—12 
in 2006—07. More information on the data received from each district is provided in Appendix A. 

The analytic models in this study estimate student achievement gains over time, comparing students who 
do and do not participate in supplemental educational services. The analyses use a fixed-effects 
approach, which operates by taking the within- student difference of the characteristics under study, and 
comparing the pre-treatment and post-treatment characteristics of each individual student. By examining 
the performance of students before and after receiving services, we are able to gain a “value-added” 
estimate of the gains associated with the services. 

Communication 

Each district provided information about their communication efforts with parents. The research team 
examined examples of submitted documents and reviewed the methods of distribution as reported by 
the districts. 
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II. FINDINGS: ELIGIBILITY FOR AND PARTICIPATION IN 
SUPPLEMENTAL EDUCATIONAL SERVICES 

Although data on Anchorage and HiUsborough are presented together in this analysis when possible, it is 
important to note that distinct features of each district, such as administration of supplemental 
educational services and supply of providers, pose challenges to a side-by-side comparison of the 
individual district’s results. While it is beyond the scope of this analysis to explore the reasons for 
possible variations in results for the districts, explanations for differences in district results might include: 

• Context: District size, demographic characteristics, prior achievement levels, and annual 
fluctuations in the number of students eligible for and participating in supplemental educational 
services might each influence the effect of supplemental educational services on student 
achievement. 

• Implementation: District capacity for implementing and monitoring supplemental educational 
services Ukely varies across districts. Differences in district implementation, such as notification 
practices and enrollment opportunities, and differences in the duration, nature, and quality of the 
various supplemental educational services programs offered in each district, including the hours 
and location of services and tutor-student ratios, may contribute to differences in results. 

Throughout the rest of the report, results for Anchorage and HiUsborough are shown together when the 
data available for each district are similar. However, with respect to achievement results, we cannot 
make direct comparisons between the two districts because the achievement measures and the grades 
tested in each district differ. 

Student Eligibility for Supplemental Educational Services 

In Anchorage, 6 percent of students were eligible for supplemental educational 
services during 2006-07. In Hillsborough, about 20 percent of students were eligible 
for these services in 2006-07. 

The percentages of students eUgible for supplemental educational services increased from 2004—05 to 
2006—07 in both districts. In Hillsborough, the percentage of students eUgible for supplemental 
educational services increased from 1 percent to 20 percent, while in Anchorage, the eUgibility rate 
increased from 1 percent to 6 percent (see Exhibit 2). 
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Exhibit 2 

Number and Percentage of Eligible Students in Grades K-12 Who Participated in 
Supplemental Educational Services, in Anchorage and Hillsborough, 

2004-05 Through 2006-07 





Number Eligible 


Percent Eligible 


Number of 
Participants 


Percentage of 
Eligible Students 
who Participated 




Anchorage, Alaska 


2004-05 


709 


1% 


30 


4% 


2005-06 


2,093 


4% 


100 


5% 


2006-07 


2,890 


6% 


308 


11% 




Hillsborough, Florida 


2004-05 


1,931 


1% 


398 


2% 


2005-06 


38,184 


18% 


3,771 


10% 


2006-07 


43,276 


20% 


6,172 


14% 



Exhibit reads: In 2004—05, 709 students were eligible for supplemental educational services in Anchorage, 
Alaska. 

Source: Anchorage School District Administrative Data, 2002—03 through 2006—07; Hillsborough School District 
Administrative Data, 2002—03 through 2006—07. 



Participation in Supplemental Educational Services 

In both districts, the percentage of eligible students participating in supplemental 
educational services increased between 2004-05 and 2006-07, reaching 11 percent 
in Anchorage and 14 percent Hillsborough in 2006-07. 

The total number of students receiving supplemental educational services in the Anchorage school 
district has been notably low.^ In 2004—05 only 30 students received services, which increased to 
308 during 2006-07 (see Exhibit 2). These low numbers are due in part to the low number of students 
eligible for these services in Anchorage and also to a low ratio of eligible students actually participating. 
During the 2006—07 school year, 11 percent of students eligible for services received them, an increase 
from the 4 percent who participated in supplemental educational services in 2004-05. 

In Hillsborough, the total number of students receiving supplemental educational services was 
considerably larger than in Anchorage. Though there were just under 400 participants in 2004—05, there 
were about 3,800 and 6,200 participants in 2005—06 and 2006—07, respectively (see Exhibit 2). This 
larger number of participants was a result of a much larger pool of eligible students. The participation 
rates of eligible students in Hillsborough were higher than in Anchorage (i.e., 10 versus 5 percent in 
2005-06, and 14 versus 11 percent in 2006-07). 
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In Anchorage, the majority of supplemental educational services participants were 
enrolled in elementary and middle school grades, while in Hillsborough almost all of 
them were in elementary school grades. 

In Anchorage, the distribution of supplemental educational services participants was fairly evenly 
distributed between elementary and middle school grades. During the 2006—07 school year, 63 and 
35 percent of participants were in grades 1-5 and 6-8, respectively (see Exhibit 3). 

The students who participated in supplemental educational services in Hillsborough were mainly 
enrolled in elementary school grades (Kindergarten though grade 5): 89 and 10 percent of all 
participants were in elementary and middle school grades, respectively, in 2006—07 (see Exhibit 3). 



Exhibit 3 

Distribution of Supplemental Educational Services Participants, by Grade Level, in 
Anchorage and Hillsborough, 2005-06 and 2006-07 



0 % 2 % 1 % 1 % 




I I Elementary Middle ^ High 



Exhibit reads: Among supplemental educational services participants in Anchorage in 2005—06, 55 percent 
of participants were in elementary school grades (K— 5), and 45 percent in middle school grades (6—8). There 
were no participants in high school grades (9-12). 

Source: Anchorage School District Administrative Data, 2002—03 through 2006—07; Hillsborough School District 
Administrative Data, 2002-03 through 2006-07. 
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The Anchorage school district served 45 percent of the students receiving 
supplemental educational services in 2006-07. In previous years, all participants 
were served by non-district providers. 

As mentioned above, the Anchorage school district began serving as a supplemental educational service 
provider in 2006—07 after it received its waiver from the U.S. Department of Education. During that 
year, the Anchorage school district served 138 students in the program, or 45 percent of aU 
308 participants. In the previous years, non-district providers served aU students who received 
supplemental educational services (see Exhibit 4). 



Exhibit 4 

Number and Percentage of Supplemental Educational Services Participants in Grades 
K-12 by Provider Type, in Anchorage, 2004-05 Through 2006-07 




^ District Provider Non-District Provider 



Exhibit reads: In Anchorage in 2006—07, 138 participants, or 45 percent of aU participants, received 
supplemental educational services from the district provider, and 170 students were served by non-district 
providers. 

Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 
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In Anchorage, five providers served students in 2005—06 and in 2006—07. Two providers served 
students in both years, so there were a total of eight providers Anchorage serving students across the 
two years. Three providers in 2005—06 and 2006—07 had fewer than 20 participants (see Exhibit 5).® 



Exhibit 5 

Number and Percentage of Students in Grades K-12 Receiving Supplemental 
Educational Services, by Provider, in Anchorage, 2005-06 and 2006-07 


Anchorage 


Provider 


2005-06 


2006-07 


# Participants 


% Participants 


# Participants 


% Participants 


Anchorage School District 


NA 


- 


138 


45% 


Private Provider A 


NA 


- 


128 


42% 


Private Provider B 


28 


28% 


19 


6% 


Private Provider C 


NA 


- 


16 


5% 


Private Provider D 


8 


8% 


7 


2% 


Private Provider E 


61 


61% 


NA 


- 


Private Provider F 


2 


2% 


NA 


- 


Private Provider G 


1 


1% 


NA 


- 


All Participants 


100 


100% 


308 


100% 


Exhibit reads: In Anchorage in 2006—07, 138 supplemental educational services participants were served 


by the Anchorage School District, representing 45 percent of all participants. 




Notes: NA indicates that the provider did not provide services during that year. 






Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 





5 Private providers are not identified by their real names. However, each letter or pair of letters represents a distinct 
provider and is used consistently throughout the report. 
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The Hillsborough school district served 971 participants (16 percent of the total) in 
2006-07. It did not offer supplemental educational services during the previous 
school years. 

As in Anchorage, the Hillsborough school district started providing supplemental educational services in 
2006—07 after it received its waiver from the U.S. Department of Education, and served 16 percent of aU 
participants that school year. The other 5,201 students were served by non-district providers, a figure 
that increased by about 40 percent from the 3,77 1 students served by these non-district providers the 
year before (see Exhibit 6). 



Exhibit 6 

Number and Percentage of Supplemental Educational Services Participants in Grades K- 
12 Who Were Served by the District Provider, in Hillsborough, 2004-05 Through 2006-07 



7.000 

6.000 

5.000 

4.000 

3,771 

3.000 

2.000 
1.000 

398 

oU=U 

2004-05 2005-06 2006-07 



6.172 



5.201 (84%) 



971 (16%) 



^ Districi Provider Non-Districi Provider 



Exhibit reads: In Hillsborough in 2006—07, 971 participants, or 16 percent of all participants, received 
supplemental educational services from the Hillsborough school district and 5,201 , or 84 percent of 
participants, received supplemental educational services from non-district providers 

Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 



The number of supplemental educational service providers was substantially higher in Hillsborough than 
in Anchorage. There were 23 service providers in Hillsborough in 2005—06 and 24 in 2006—07, 
respectively (see Exhibit 7). In Hillsborough only one provider served fewer than 20 students in 2005— 
06 and only three providers served fewer than 20 students in 2006—07. 

It is important to note that, in Hillsborough in 2005—06 and 2006—07, some participants received 
services from more than one provider.'^ 



Adding the number of students served by each provider in 2006—07 gives a total figure of 6,246, which is higher than 
the 6,172 students participating in the program during that year. This indicates that some students had more than one 
provider. 
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Exhibit 7 








Number and Percentage of Students in Grades K-12 Receiving Supplemental 


Educational Services, by Provider, in Hillsborough, 2005-06 and 2006-07 


Hillsborough 




2005- 


-06 


2006-07 


Provider Name 


N 


% 


N 


% 


Hillsborough County Public Schools — Academy For 
Success (District provider) 


NA 


- 


971 


16% 


Private Provider H 


637 


17% 


1,226 


20% 


Private Provider I 


739 


19% 


820 


13% 


Private Provider J 


121 


3% 


787 


13% 


Private Provider K 


309 


8% 


332 


5% 


Private Provider L 


328 


9% 


312 


5% 


Private Provider M 


NA 


- 


298 


5% 


Private Provider B 


NA 


- 


276 


0% 


Private Provider N 


336 


9% 


214 


3% 


Private Provider 0 


183 


5% 


150 


2% 


Private Provider P 


72 


2% 


148 


2% 


Private Provider Q 


40 


1% 


121 


2% 


Private Provider R 


308 


8% 


106 


2% 


Private Provider S 


NA 


- 


84 


1% 


Private Provider T 


66 


2% 


72 


1% 


Private Provider A 


NA 


- 


71 


1% 


Private Provider U 


75 


2% 


67 


1% 


Private Provider V 


28 


<1% 


61 


1% 


Private Provider W 


NA 


- 


54 


<1% 


Private Provider Y 


97 


3% 


36 


<1% 


Private Provider Z 


NA 


- 


29 


<1% 


Private Provider AA 


NA 


- 


5 


<1% 


Private Provider AB 


NA 


- 


4 


<1% 


Private Provider AC 


NA 


- 


2 


<1% 


Private Provider AD 


135 


4% 


NA 


- 


Private Provider AE 


80 


2% 


NA 


- 


Private Provider AF 


56 


2% 


NA 


- 


Private Provider AG 


54 


1% 


NA 


- 


Private Provider AH 


54 


1% 


NA 


- 


Private Provider Al 


43 


1% 


NA 


- 


Private Provider AJ 


40 


1% 


NA 


- 


Private Provider AK 


32 


<1% 


NA 


- 


Private Provider AL 


17 


<1% 


NA 


- 


Total 


3,850 


100% 


6,246 


100% 


Exhibit reads: In Hillsborough in 2006—07, 971 supplemental educational services participants 


were 


served by the district, representing 1 6 percent of all participants. 








Note: NA indicates that the provider did not provide services during that year. 

Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Characteristics of Eligible and Participating Students 

African-American and Alaska Native students constituted the largest ethnic groups 
of all students receiving supplemental educational services in Anchorage. Students 
served by the Anchorage school district were more likely to be Alaska Native or 
Hispanic, while students served by non-district providers were more likely to be 
African-American or Asian. 



In Anchorage, African-American and Alaska Native students constituted the largest ethnic groups of all 
participants, representing 21 and 20 percent of the total, respectively. Other racial-ethnic groups that 
each comprised over ten percent of participants were white, Asian, multi-racial, and Hispanic students 
(see Exhibit 8). 



Exhibit 8 

Demographic Characteristics of Supplemental Educational Services Participants in 
Grades K-12, by Provider Type, in Anchorage, 2006-07 




All Eligible 


All Participants 


Anchorage 
School District 


Non-District 

Providers 


Distribution by Race-Ethnicity 


% African-American 


14** 


21** 


15* 


25* 


% Aiaska Native 


25** 


20** 


22 


18 


% White 


16 


17 


16 


17 


% Asian 


25** 


16** 


11* 


19* 


% Multi-Raciai 


11** 


14** 


16 


13 


% Hispanic 


8** 


12** 


20* 


6* 


% American Indian 


1 


1 


1 


1 


Totai 


100% 


100% 


100% 


100% 


N 


2,890 


308 


138 


170 


Exhibit reads: In Anchorage in 2006—07, 14 percent of students eligible for supplemental educational 
services were African-American and 21 percent of aU participants were African-American. Fifteen percent of 
supplemental educational services participants served by the district were African-American, while 25 percent 
of participants served by non-district providers were African-American. 

Note: * indicates a significant difference between district and non-district providers at the .05 level. ** Indicates a 
significant difference between aU eligible students and all participants at the .05 level. 

Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 



There were statistically significant differences in the ethnic background of students served by district and 
non-district providers in Anchorage. The Anchorage school district tended to serve more Alaska Native 
and Hispanic students, while non-district providers were more likely to serve African-American and 
Asian students. The differences in the percentages of white and multi-racial students across these 
providers were less pronounced, though still statistically significant. 

Moreover, African-American students — and to a lesser degree white, multi-racial and Hispanic 
students — ^were more likely to participate in supplemental educational services than were Alaska Native 
and Asian students. Appendix B presents participation rates by ethnic group in Anchorage. 

Data on English proficiency and special education status were not available for Anchorage. 
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In Hillsborough, Hispanic and African-American students represented more than 
80 percent of all students receiving supplemental educational services. 



In HiUsborough, Hispanic and African-American students constituted the vast majority, over 80 percent, 
of participants. The differences in student ethnic background across district and non-district providers 
were less pronounced in HiUsborough than in Anchorage. African-American and white students were 
more Ukely to choose the district provider for services, while Hispanic students were more Ukely to 
choose a non-district provider. Hispanic students were also more Ukely to participate in supplemental 
educational services than were white students (see Exhibit 9). Appendix B presents participation rates by 
ethnic group in HiUsborough. 

About 45 percent of aU students receiving supplemental educational services in HiUsborough were 
Umited EngUsh proficient. The pool of students receiving services from the district had a sUghdy lower 
percentage of Umited EngUsh proficient students than those served by non-district providers (42 versus 
46 percent, respectively). In addition, 21 percent of supplemental educational services participants were 
special education students. Students served by the HiUsborough school district were somewhat more 
Ukely to be special education students than those served by non-district providers (see Exhibit 9). 





Exhibit 9 






Demographics Characteristics of Supplemental Educational Services Participants in 
Grades K-12, by Provider Type, in Hillsborough, 2006-07 




All Eligible 


All Participants 


Hillsborough 
School District 


Non-District 

Providers 


Distribution by Race-Ethnicity 


% African-American 


35** 


33** 


35 


33 


% White 


17** 


14** 


15 


13 


% Asian 


1 


1 


1 


1 


% Multi-Raciai 


4 


4 


3* 


4* 


% Hispanic 


43** 


48** 


45* 


49* 


% American Indian 


<1 


<1 


<1 


<1 


Students with Speciai Needs 


% Limited English Proficient 


38 


45 


42* 


46* 


% Speciai Education 


18 


21 


24* 


20* 


N 


43,276 


6,172 


971 


5,201 


Exhibit reads: In HiUsborough in 2006—07, 35 percent of students eUgible for supplemental educational 
services were African-American and 33 percent of all participants were African-American. Thirty-five 
percent of supplemental educational services participants served by the district were African-American, whUe 
33 percent of participants served by non-district providers were African-American. 


Note: * indicates a significant difference between district and non-district providers at the .05 level. ** 
significant difference between aH eligible students and all participants at the .05 level. 


indicates a 


Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Hours and Subject of Tutoring 

Participants in the program received an average of 28 and 22 hours of supplemental 
educational services in Anchorage and Hillsborough during 2006-07, respectively. 

In 2006—07, the average number of hours of supplemental educational services was 28 hours in 
Anchorage. Students served by the Anchorage school district received, on average, six (or 24 percent) 
more service hours than those who were served by non-district providers. In the case of Hillsborough, 
the average number of hours of tutoring and the difference between district and non-district providers 
was lower than in Anchorage. In Hillsborough, participants received an average of 22 hours of 
supplemental educational services. Non-district providers served students, on average, for about one 
hour more than the district provider (see Exhibit 10). 



Exhibit 10 

Average Number of Hours of Supplemental Educational Services Received by 
Participants in Grades K-12, by Provider, in Anchorage and Hillsborough, 2006-07 



60 

50 

40 

30 



20 
10 
0 

Anchorage Hillsborough 



^ District Provider Q Non-District Providers Total 



Exhibit reads: In Anchorage in 2006—07, the district provider delivered an average of 31 hours of 
supplemental educational services per student, while non-district providers delivered an average of 25 hours 
of supplemental educational services per student. The average number of hours for services for all 
participating students was 28. 

Note: * Indicates a statistically significant difference between the district provider and non-district providers at the 
.05 level. 

Source; Anchorage School District Administrative Data, 2002—03 through 2006—07; Hillsborough School District 
Administrative Data, 2002—03 through 2006—07. 
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In Anchorage during 2006-07, 44 percent of students (135 students) received only supplemental 
educational services in reading, while 17 percent (51 students) received only services in mathematics. 
Another 23 percent (72 students) received tutoring in both subjects. There are 50 supplemental 
educational services participants in Anchorage for whom information on the subject of tutoring is 
missing (see Appendix C). Students served by the Anchorage school district were more likely to receive 
supplemental educational services in both subjects (39 percent of district participants), compared to 
students served by non-district providers (11 percent). See Appendix C for further details. 

In Hillsborough, about 79 percent of all participants (4,857 students) received supplemental educational 
services only in reading during the 2006—07 school year. Twenty-one percent of students (1,283 
students) received supplemental educational services only in mathematics. This means that in 
Hillsborough only 32 students received supplemental educational services in both subjects (see Appendix 
C). In terms of the differences across district and non-district providers, students served by the 
Hillsborough school district were more likely to receive only mathematics services (31 percent of 
students served by the district compared to 19 percent of students served by non-district providers 
received support in mathematics only). See Appendix C for further details. 



Chapter II 



17 





III. FINDINGS: STUDENT ACHIEVEMENT 



Analytic Approach 

The analytic approach follows the methods used for a previous analysis of SES and student achievement 
in nine large urban school districts that was conducted as part of the National Longitudinal Study of No 
Child Left Behind (Zimmer et ah, 2006). For Hillsborough, we use longitudinal, student-level data for aU 
students in grades 1—12 from 2002—03 through 2004—05, grades 2—12 in 2005—06, and grades 3—12 in 
2006-07. The achievement test scores come from Florida’s Comprehensive Assessment Test (FCAT). 

In Hillsborough, information on participation in supplemental educational services, type of provider, 
number of hours of services, and subject of services was available for 2005—06 and 2006—07. 

In Anchorage, we used longitudinal, student level data for all students in grades 3—9 in 2004—05, and 
grades 3—10 in 2005—06 and 2006—07. Unfortunately, before 2004—05 only achievement data for 
students in grades 3, 6, and 8 were available. The achievement data in Anchorage comes from Alaska’s 
Standards Based Assessment (SBAs) for 2004—05 onwards, while previous achievement data come from 
the Alaska Benchmark Exam. 

The outcomes for both districts are rank-based z-scores that indicate a student’s position in the test 
score distribution for his or her grade level. For example, a score of 0.5 indicates that the student scored 
one half of a standard deviation above the district mean. It is important to note that gains on these 
metrics do not indicate a developmental improvement, such as a five-point gain, rather they represent a 
student’s movement within the test score distribution relative to his or her classmates.^ 

The analytic models estimate student achievement gains over time, comparing students who do and do 
not participate in supplemental educational services. By combining longitudinal data and student level 
fixed-effects, we can isolate the gains associated with participation in supplemental educational services 
by parceling out the effect of time invariant student characteristics, such as parent education, income or 
underlying student ability or motivation. The fixed-effects approach operates by taking the 
within- student difference of the characteristics under study (e.g., eligibility for supplemental educational 
services, participation in services, and provider chosen), and comparing the pre-treatment and 
post-treatment characteristics of each individual student. By examining the performance of students 
before and after receiving services, we are able to gain a “value-added” estimate of the services. Further, 
by including a term for each individual student in the analytic model, we further guard against selection 
bias that could result if students choosing services were different from others in some immeasurable 
way. This student-level fixed-effect essentially controls for the characteristics of each student that may 
otherwise affect their achievement levels. For example, students who choose a program may be more 
motivated than others, or may have other unmeasured traits (e.g., higher parental involvement or 
education) that may make them more likely to achieve at higher levels even in absence of the program, 
which can lead to biased results. Further, students who are more motivated or who may have greater 
levels of innate ability (beyond what may be reflected in a standardized test score) may benefit more 
from services. It is possible that selection bias may lead to either overestimation or underestimation of 
the true gains associated with the program. 



^ This is analogous to a national percentile rank where the mean is zero rather than 50 percent, and the population is the 
district rather than the nation. 
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The gains model is: 

(1) Ait — Ait-i= cc Eligibleit + X Parti cipationk + pt + 6gt + Sk 



Where-. 

Ait — Ak-i is the achievement gain of student i from year t— 1 to year t 
Eligibleit is the eligibility status of student i in year t 
Participationit is the participation status of student i in year t 
pt represents the individual student’s fixed effect 
0gt represents the grade by year fixed-effects 
Sit represents the random error 

An additional variation on the model includes interactions for participation and student characteristics, 
which evaluates whether certain groups of students may benefit more from supplemental educational 
services. These models build on equation 1 by adding an interaction term for the participation by a 
student group, such as limited English proficient (LEP) students. A significant positive coefficient for 
LEP students by participation interaction term would indicate that LEP participants showed significant 
gains in achievement.* 

In order to evaluate the gains associated with various characteristics of supplemental educational 
services, such as provider type, hours, etc., we estimate a separate model for each characteristic. We 
estimate a total of 7 models for each district to evaluate the effect of the following variables: overall 
participation; first year of participation as compared to multiple years of participation; district versus 
non-district provider; individual providers; hours of services received; and subject of tutoring. These 
models substitute the full set of options for the participation variable, such as all providers, and estimate 
the gains associated with each separately. For example, the model for provider type is as follows: 

(2) Ak — Ait-i = a EUgibleit + X District Provident + ^ Non- District Provident + pt + 6gt + Sk 

The coefficient for the district provider variable indicates the achievement gain for students served by 
the district’s supplemental educational services program, while the non-district coefficient indicates the 
gain for participants served by non-district providers.^ 

The analyses of district providers versus non-district providers are restricted to those students who 
received supplemental educational services during 2006-07, the year of the waiver. That is, students who 
received SES services in 2004-05 or 2005-06 but not in 2006-07 were excluded from these analyses. 

As a result of the analytic approach described above, the results of the achievement analyses reported in 
this chapter are based on a subset of the full student populations in each district. While earlier chapters 



* The interpretation of these interaction terms is complicated; they are designed to indicate whether a particular group of 
students is making significant gains. For example, to examine gains for Hispanic students, we use an interaction term to 
indicate the specific gain for Hispanic participants, while the comparison group is non-Hispanic, non-participants. 
Therefore the model estimates the gains for aU groups, to show which groups made significant gains and which did not. 
The significance test measures whether the effect of participation plus the effect of being Hispanic is significantly 
different from 2ero. 

5 Note that this approach does not directly compare the gains of students served by district versus non-district providers; 
it tests whether students served by the district make gains that are significantly greater than 2ero, and concurrently tests 
whether the gains made by students served by non-district providers were significantly greater than 2ero. The relative 
sEe of the gains can be seen by comparing the coefficients for each provider type, which show the average gains in 
standard deviation units. 



Chapter III 



20 





included aU students for whom we had information on eligibility and participation information, this 
chapter includes only students in tested grades. The analytic approach further requires that students 
have achievement scores for at least two time points in either reading or mathematics in order to 
construct a gain score. Finally, students who repeated a grade were not included in the analyses. Given 
that the outcome measure is designed to measure a student’s progress in relation to the rest of the 
students in his or her grade level, including a student who has repeated a grade inherentiy leads to 
comparing an on-track student to a student who is one year older and has taken the same test twice. For 
example, if we examine a cohort of students in fifth grade in 2004—05, we look at their outcome as the 
difference between their reading score in the prior year as fourth graders and their reading score in 
2004—05 as fifth graders, both of which are standardi 2 ed to the full population of test takers of that 
grade level in that year. Any fifth graders who repeated fifth grade in that year may have much larger 
gains, given they have taken the same test in both years, and the non-repeaters may appear to be doing 
less well. 

In addition to describing the caveats to the analytical approach above, is important to note that because 
the achievement measures as well as the grades tested in each district differ, we are limited in our ability 
to make direct comparisons between achievement results in Anchorage and Hillsborough. 
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Prior Achievement Levels by Student Eligibility for Supplemental 
Educational Services 



In both Anchorage and Hillsborough, students who were eligible for supplemental 
educational services had significantly lower levels of prior achievement in 
mathematics and reading than students who were not eligible. 

Comparing the previous year’s achievement scores of students who were eligible for supplemental 
educational services (students from low-income families who attend Tide I schools that did not make 
AYP for the third year) to the scores of students in the district who were not eligible shows whether 
students who were eligible for supplemental educational services were more likely to have lower levels of 
prior achievement. As one would expect, eligible students in both districts had lower prior achievement 
scores than ineligible students, in both reading and mathematics. 

As mentioned in the previous section, scale scores have been standardized within grade (to a mean of 
zero, and a standard deviation of one) in order to make comparisons across grades more meaningful. In 
Anchorage, students who were eligible for supplemental educational services in 2006—07 scored 0.57 and 
0.69 standard deviations below the overall district mean on the prior year’s mathematics and reading test, 
respectively. In contrast, ineligible students were just above the overall district mean (see Exhibit 11). 



Exhibit 11 

Prior Achievement Levels (in 2005-06) on the SBA of Students in Grades 3-10 by 
2006-07 Supplemental Educational Services Eligibility, in Anchorage 



Anchorage 




Mathematics 


Reading 


Ineligible in 2006-07 


0.04 


0.05 


Eligible in 2006-07 


-0.57* 


-0.69* 


n 


27,430 


27,376 



Exhibit reads: In Anchorage, students who were not eligible for supplemental educational services in 
2006—07 had levels of prior mathematics achievement in 2005—06 of about 0.04 standard deviations above 
the district average, while students who were eligible had prior mathematics achievement levels 0.57 standard 
deviations below the district average, a significant difference. 

Notes: Means are for all students with at least one gain score and who are therefore included in the regression analyses. 
Scores reported are rank-based z-scores. 

* indicates a significant difference between eligible and ineligible students at the .05 level. 

Source; Anchorage School District Administrative Data, 2005—06 through 2006—07. 
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The prior achievement gap between eligible and ineligible students was much smaller in Hillsborough 
than in Anchorage, possibly a result of the much higher percentage of eligible students in Hillsborough. 
StiU, as in Anchorage, students eligible for supplemental educational services in Hillsborough had lower 
achievement levels than ineligible students. Students who were eligible for supplemental educational 
services in 2006—07 scored 0.10 and 0.11 standard deviations below the overall district mean on the prior 
year’s mathematics and reading test, respectively. On the other hand, ineligible students scored 
0.02 standard deviations above the district mean on the prior year’s mathematics and reading tests (see 
Exhibit 12). 



Exhibit 12 

Prior Achievement Levels (in 2005-06) on the FCAT of Students in Grades 1-12 by 
2006-07 Supplemental Educational Services Eligibility, in Hillsborough 



Hillsborough 




Mathematics 


Reading 


Ineligible in 2006-07 


0.02 


0.02 


Eligible in 2006-07 


-0.10* 


-0.11* 


n 


195,432 


195,432 



Exhibit reads: In Hillsborough, students who were not eligible for supplemental educational services in 
2006—07 had levels of prior mathematics achievement in 2005—06 of about 0.02 standard deviations above 
the district average, while students who were eligible had prior mathematics achievement levels 0.10 standard 
deviations below the district average, a significant difference. 

Notes: Means are for all students with at least one gain score and who are therefore included in the regression analyses. 
Scores reported are rank-based 2 -scores. 

* indicates a significant difference between eligible and ineligible students at the .05 level. 

Source: Hillsborough School District Administrative Data, 2005—06 through 2006—07. 



Chapter III 



23 



Prior Achievement Levels by Student Participation in Supplemental 
Educational Services 



Students who participated in supplemental educational services had lower prior 
achievement levels than eligible non-participants in Anchorage and Hillsborough 

In Anchorage, students who participated in supplemental educational services in 2006-07 performed 
worse in mathematics and reading than eligible non-participants in 2005—06. Participating students 
scored 0.95 and 1.03 standard deviations below the Anchorage district mean in mathematics and reading, 
respectively, while eligible non-participants scored 0.52 and 0.64 standard deviations below this overall 
mean (see Exhibit 13). 



Exhibit 13 

Prior Achievement Levels (in 2005-06) on the Benchmark of Students in Grades 3-10 by 
2006-07 Supplemental Educational Services Participation, in Anchorage 



Anchorage 




Mathematics 


Reading 


Participants in 2006-07 


-0.95* 


-1.03* 


Eiigibie Non-Participants in 2006-07 


-0.52 


-0.64 


Students who where ineiigibie in 2006-07 


0.04 


0.05 


n 


1,474 


1,460 



Exhibit reads: In Anchorage, students who received supplemental educational services in 2006—07 had 
levels of prior mathematics achievement in 2005—06 of about 0.95 standard deviations below the district 
average. 

Notes: Means are for all students with at least one gain score and who are therefore included in the regression analyses. 
Scores reported are rank-based 2 -scores. 

* indicates a significant difference between eligible non-participants and participants at the .05 level. 

Source: Anchorage School District Administrative Data, 2005—06 through 2006—07. 
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In HiUsborough the pattern was similar, with participants having lower prior achievement than eligible 
non-participants. The academic achievement of students who participated in supplemental educational 
services in 2006-07 was equal to 0.18 standard deviations below the district mean in 2005-06 across 
both subjects (see Exhibit 14). During that year, the performance of eligible non-participants ranged 
from 0.09 to 0.10 standard deviations below the district mean in mathematics and reading, respectively. 
This indicates that eligible non-participants scored between 0.08 to 0.09 standard deviations above 
participants on prior achievement tests. 



Exhibit 14 

Prior Achievement Levels (in 2005-06) on the FCAT of Students in Grades 1-12 by 
2006-07 Supplemental Educational Services Participation, in Hillsborough 



Hillsborough 




Mathematics 


Reading 


Participants in 2006-07 


-0.18* 


-0.18* 


Eiigibie Non-Participants in 2006-07 


-0.09 


-0.10 


Students who where ineligibie in 2006-07 


0.02 


0.02 


n 


31,227 


31,227 



Exhibit reads: In Hillsborough, students who received supplemental educational services in 2006—07 had 
levels of mathematics achievement in 2005—06 of about 0.18 standard deviations below the district average. 

Notes: Means are for all students with at least one gain score and who are therefore included in the regression analyses. 
Scores reported are rank-based 2 -scores. 

* indicates a significant difference between eligible non-participants and participants at the .05 level. 

Source: Hillsborough School District Administrative Data, 2005—06 through 2006—07. 
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Prior Achievement Levels by Provider Type 

In Hillsborough, participants served by the school district had higher prior 
achievement levels than students served by other providers. In Anchorage, prior 
student achievement across district and non-district providers was statistically 
indistinguishable. 



In Anchorage, there was no clear pattern of prior achievement levels in 2005—06 between students 
served by the district and those served by non-district providers. AH of these students scored around 1 
standard deviation below the district mean in 2005—06 in mathematics and reading (see Exhibit 15). 



Exhibit 15 

Prior Achievement Levels (in 2005-06) on the Benchmark of Students in Grades 3 
Through 10 by 2006-07 Provider Type, in Anchorage 



Anchorage 




Mathematics 


Reading 


District Provider in 2006-07 


-1.05 


-1.06 


Non-District Providers in 2006-07 


-0.89 


-1.01 


n 


181 


182 



Exhibit reads: In Anchorage, students who were served in 2006-07 by the district provider had an average 
level of mathematics achievement of 1.05 standard deviations below the district average in 2005—06, while 
students served by non-district providers had prior mathematics achievement levels during that year of 0.89 
standard deviations below the district average. Nevertheless, this difference was not statistically significant. 

Notes: Means are for all students with at least one gain score and who are therefore included in the regression analyses. 
Scores reported are rank-based 2 -scores. 

* indicates a significant difference between students served by district versus non-district providers at the .05 level. 
Source: Anchorage School District Administrative Data, 2005—06 through 2006—07. 
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In HiUsborough, participants served by the school district in 2006—07 scored sUghtiy higher on prior 
achievement tests than those served by non-district providers. For instance, students served by non- 
district providers in 2006—07 scored, on average, 0.19 standard deviations below the district mean in 
mathematics and reading in 2005—06, while students served by the Hillsborough school district scored 
0.14 and 0.13 standard deviations below the district mean in mathematics and reading, respectively (see 
Exhibit 16). 



Exhibit 16 

Prior Achievement Levels (in 2005-06) on the FCAT of Students in Grades 1-12 by 

2006-07 Provider Type, in Hillsborough 



Hillsborough 




Mathematics 


Reading 


District Provider in 2006-07 


-0.14 


-0.13 


Non-District Providers in 2006-07 


-0.19* 


-0.19* 


n 


4,192 


4,192 



Exhibit reads: In Hillsborough, students who were served by the district provider in 2006—07 had an 
average level of mathematics achievement of 0.19 standard deviations below the district average in 2005—06, 
while students served by non-district providers had prior mathematics achievement levels during that year of 
0.14 standard deviations below the district average, a significant difference. 

Notes: Means are for all students with at least one gain score and who are therefore included in the regression analyses. 
Scores reported are rank-based 2 -scores. 

* indicates a significant difference between students served by district versus non-district providers at the .05 level. 
Source: Hillsborough School District Administrative Data, 2005—06 through 2006—07. 
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Supplemental Educational Services Participation and Student 
Achievement 

The following results are based on the student-level fixed-effects models described in the analytic 
approach section. AU models include an indicator for student eUgibiUty for services, plus grade, year, and 
grade-by-year fixed-effects. Note that the coefficients presented in these regression tables always compare 
the achievement gains of participants to the gain of eligible non-participants, using standardized scale scores 
(i.e., a z-score metric). This means that each coefficient indicates whether participants experienced a larger or 
smaller achievement gain (measured in standard deviations) than the one of eligible non-participants, and 
whether this difference was statistically significandy different or not. 

Of the students in this study, only those who received supplemental educational 
services in mathematics in Hillsborough experienced larger academic gains (in 
mathematics) than eligible non-participants. No gains were found in Anchorage, but 
the participant group was extremely small which makes it difficult to detect 
significant effects. 

On average, students receiving supplemental educational services in Anchorage did not experience larger 
gains in academic performance than eligible non-participants (see Exhibit The coefficients 

associated with the overall gains in mathematics and reading were not statistically significant.!^ Note that 
a negative coefficient does not necessarily imply an achievement decline of the participant group. It 
indicates that students receiving supplemental educational services experienced smaller gains (or 
potentially, larger declines) than eligible non-participants. It is a relative, not an absolute, measure of 
performance. 



Exhibit 17 

Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 
by Participation in Supplemental Educational Services, in Anchorage, 

2002-03 Through 2006-07 





Mathematics 


Reading 


Gains 


n 


Gains 


n 




Overall Effect 


-0.05 


221 


0.01 


258 


First Year Effect 


-0.04 


215 


0.01 


255 



Exhibit reads: In Anchorage, students who participated in supplemental educational services in 
mathematics showed no significant differences in mathematics achievement gains from those of eligible non- 
participants. 

Notes: Scores reported are rank-based z-scores. Multiple Year Effect results not included due to small n size (6 

participants total for 2005-06 and 2006-07). N is the number of student observations in treatment contributing 
to the estimate. All models controlled for student eligibility for services. * indicates gains that were statistically 
different form the gains of eligible non-participants at the .05 level. 

Source; Anchorage School District Administrative Data, 2002—03 through 2006—07. 



!*! Robust standard errors of ail fixed-effects regressions are presented in Appendix E. 

!! In statistical terms, the power to reject the hypothesis of no effect (nuU hypothesis) depends on the number of 
observations available. The larger the treatment and control group, the higher the power is to reject the null hypothesis. 
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In HiUsborough, students receiving supplemental educational services experienced higher achievement 
gains in mathematics than eligible students who chose not to participate. This difference was equal to 
0.05 standard deviations in this subject; this analysis controls for the subject in which a student received 
services (see Exhibit 18). No statistically significant differences were observed between participants and 
eligible non-participants in reading. 

In HiUsborough, when the effect of the first year of program participation is isolated from the 
subsequent years, participants experienced larger gains in mathematics than eUgible non-participants only 
during the first year of the program. Again, no statisticaUy significant differences were observed in 
reading. 



Exhibit 18 

Achievement Gains on the FCAT of Students in Grades 1-12 by Participation 
in Supplemental Educational Services, in Hillsborough, 

2002-03 Through 2006-07 





Mathematics 


Reading 


Gains 


n 


Gains 


n 




Overall Effect 


0.05* 


1,966 


-0.001 


7,288 




First Year Effect 


0.04* 


1,683 


-0.0001 


6,092 


Multiple Year Effect 


0.07 


283 


-0.01 


1,196 



Exhibit reads: In Hillsborough, students who participated in supplemental educational services 
demonstrated greater gains in mathematics achievement — improving by .05 standard deviations — relative to 
eligible non-participants, a statistically significant difference. 

Notes: Scores reported are rank-based 2 -scores. N is the number of student observations in treatment contributing to 
the estimate. All models controlled for student eligibility for services. 

* indicates gains that were statistically different form the gains of eligible non-participants at the .05 level. 
Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Gains in Student Achievement by Provider 

In the aggregate, only students served by non-district providers in Anchorage and 
Hillsborough experienced larger academic gains in mathematics compared with 
eligible non-participants. 



On average, students who received services from non-district providers in Anchorage experienced 
statistically significant achievement gains in comparison to eligible non-participants (see Exhibit 19). 



Exhibit 19 

Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 
by Provider Type, in Anchorage, 2002-03 Through 2006-07 





Mathematics 


Reading 




Gains 


n 


Gains 


n 


District Provider 


-0.17 


67 


-0.11 


83 


Non-District Providers 


0.18* 


62 


0.12 


90 



Exhibit reads: In Anchorage, students who received supplemental educational services in mathematics from 
the district provider showed no significant difference in mathematics achievement gains from those of 
eligible non-participants. Students who were served by non-district providers demonstrated greater gains in 
mathematics achievement, on average — improving by 0.18 standard deviations — relative to eligible non- 
participants, a statistically significant difference. 

Notes: Scores reported are rank-based 2 -scores. AH models controlled for student eligibility for services. N is the 

number of student observations in treatment contributing to the estimate. * indicates gains that were statistically 
different from the gains of eligible non-participants at the .05 level. 

Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 
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In Anchorage, no provider showed achievement gains that were significantiy higher than the gains of 
eligible non-participants. One provider in Anchorage had gains in mathematics that were higher than 
the average gain across aU participants. This provider experienced a gain of 0.17 standard deviations 
above the performance of eligible non-participants, and served 46 students for an average of 31 hours. 
All the other providers in Anchorage experienced mathematics achievement gains statistically 
indistinguishable from the average coefficient across aU participants (—0.05 standard deviations) 

(see Exhibit 20). The small sample size of students served by each provider makes it difficult to detect 
significant gains. 



Exhibit 20 

Mathematics Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 by 
Eligibility for and Participation in Supplemental Educational Services, by Provider, in 

Anchorage, 2002-03 Through 2006-07 



Provider Name 


Number of 
Participants 


Average Hours per 
Student 


Mathematics 


Average Across All Providers 


-0.05 



Providers Showing No Significant Effects 



Private Provider A 


46 


31 


0.17 


Anchorage School District 


67 


29 


-0.17 


Other Providers 


16 


28 


0.21 


Total Number of Students 


129 







Exhibit reads: In Anchorage, Private Provider A provided supplemental educational services to 46 students 
in 2006—07, with an average of 31 hours of services per student. Students served by Private Provider A 
showed a statistically significant difference in mathematics achievement gains compared with the average 
participant gain, but not with respect to eligible non-participants in the district. 

Notes: Scores reported are rank-based z-scores. The “Other Providers” category includes aU providers serving less than 20 
students in mathematics. All models controlled for student eligibility for services. 

* indicates that the provider experienced gains that were statistically different from the gains of eligible non- 
participants at the .05 level. Scores reported are rank-based z-scores. Shaded bars indicate that the provider 
experienced gains that were statistically different from the district average for aU participants. In the next exhibit, 
no bars are shaded, indicating that there are no statistically significant differences from the district average. 

Source; Anchorage School District Administrative Data, 2002—03 through 2006—07. 
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In Anchorage, no provider — including the Anchorage school district — had statistically significant gains 
in reading above or below the group average gain (see Exhibit 21). In other words, none of these 
performance gains was statistically different from the overall average participant gain of 0.01 standard 
deviations above that of eligible non-participants (see Exhibit 17). These providers served between 20 
and 70 students in reading for an average of 26 to 33 hours. 



Exhibit 21 

Reading Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 by 
Eligibility for and Participation in Supplemental Educational Services, by Provider, in 

Anchorage, 2002-03 Through 2006-07 



Provider Name 


Number of Participants 


Average Hours per 
Student 


Reading 


Average Across All Providers 


0.01 



Providers Showing No Significant Effects 



Private Provider A 


70 


27 


0.11 


Anchorage Schooi District 


83 


33 


-0.11 


Other Providers 


20 


26 


0.14 


Totai Number of Students 


173 







Exhibit reads: In Anchorage, Private Provider A showed achievement gains for participating students that were 
not significantly different than gains for eligible non-participants. This gain was also not significantly larger than 
the average participant gain in the district. 

Notes: Shading indicates that the provider experienced gains that were statistically different from the district average. 

* indicates that the provider experienced gains that were statistically different from the gains of eligible non- 
participants at the .05 level. Scores reported are rank-based z-scores. The “Other Providers” other category 
includes all providers serving less than 20 students in reading. All models controlled for student eligibility for 
services. 

Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 
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In Hillsborough, only students served by non-district providers experienced higher 
academic gains in mathematics than eligible non-participants. 



In HiUsborough, only students served by non-district providers showed statistically significant 
achievement gains in mathematics, 0.08 standard deviations, compared to eligible non-participants (see 
Exhibit 22). Consistent with the overall program effects observed in Hillsborough (Exhibit 18), these 
results indicate no statistically significant difference between participants and eligible non-participants in 
reading. 

In Hillsborough a hybrid category existed, composed of students who were served simultaneously by the 
district and non-district providers in 2006—07. The data show that this hybrid group of participants did 
not experience significantiy different academic gains than eligible non-participants. 



Exhibit 22 

Achievement Gains on the FCAT of Students in Grades 1-12 
by Provider Type, in Hillsborough, 2002-03 through 2006-07 





Mathematics 


Reading 




Gains 


n 


Gains 


n 


District Provider 


0.05 


234 


-0.02 


447 


Non-District Providers 


0.08* 


743 


-0.01 


2,503 


District and Non-District Providers® 


★ ★ 


18 


0.22 


50 



Exhibit reads: In Hillsborough, students who received supplemental educational services in mathematics 
from the district provider showed no significant difference in mathematics achievement gains from those of 
eligible non-participants. Students who were served by non-district providers demonstrated greater gains in 
mathematics achievement — improving by 0.08 standard deviations — relative to eligible non-participants, a 
statistically significant difference. 

Notes: Scores reported are rank-based 2 -scores. N is the number of student observations in treatment contributing to 
the estimate. All models controlled for student eligibility for services. 

* indicates that the provider experienced gains that were statistically different from the gains of eligible non- 
participants at the .05 level. 

** indicates that the data are not reported due to small sample si 2 e (n<20 students). 

“ Some students in Hillsborough were served in 2006—07 simultaneously by the district provider and non-district 
providers, creating a hybrid group. 

Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Out of the 10 providers in HiUsborough that served more than 20 participants in mathematics during the 
2006—07 school year, one provider showed achievement gains that were significantiy higher than the 
gains of eligible non-participants (provider S, noted with an asterisk) (see Exhibit 23). However, no 
provider had achievement gains in mathematics that were statistically greater than the average gain across 
aU participants (0.05). These 10 providers served between 20 and 234 students in mathematics in 2006— 
07 for an average number of hours that ranged from 9 to 35 hours. One of these providers was the 
district, which served 234 participants in mathematics for an average of 20 hours. 





Exhibit 23 




Mathematics Achievement Gains on the FCAT of Students in Grades 1-12 by Eligibiiity for 


and Participation in Suppiemental Educational Services, by Provider, in Hillsborough, 

2002-03 through 2006-07 


Provider Name 


Number of 
Participants 


Average Hours per 
Student 


Mathematics 


Average Across All Providers 


0.05 


Providers Showing Significant Gains 


Private Provider S 


20 


27 


0.32* 


Providers Showing No Significant Gains 


Private Provider W 


22 


9 


0.14 


Private Provider Q 


49 


24 


0.08 


Private Provider N 


40 


35 


0.08 


Private Provider I 


158 


18 


0.07 


Private Provider H 


213 


24 


0.05 


Hiiisborough Schooi District 


234 


21 


0.05 


Private Provider L 


55 


16 


0.05 


Private Provider K 


52 


19 


0.02 


Private Provider B 


44 


16 


-0.04 


Other Providers 


128 


24 


0.14 


Totai Number of Students 


995 






Exhibit reads: In Hillsborough, one private provider (S) showed achievement gains for participating students 
that were significantly larger than gains for eligible non-participants. However, these gains were not statistically 
larger than the average effect size across all supplemental educational service providers. 


Notes: Scores reported are rank-based z-scores. The “Other Providers” category includes all providers serving less than 
20 students in mathematics. All models controlled for student eligibility for services. 


* indicates that the provider experienced gains that were statistically different from the gains of eligible non- 
participants at the .05 level. 

Shaded bars indicate that the provider experienced gains that were statistically different from the district average. 
In this exhibit, no bars are shaded, indicating that there are no statistically significant differences from the district 


average. 








Source: Hillsborough School District Administrative Data, 2002 


-03 through 2006-07. 





In reading, 19 providers in Hillsborough served more than 20 participants during the 2006—07 school 
year. No provider statistically performed above the overall average participant gain in the district or 
above eligible non-participants (see Exhibit 24). The reading achievement gains ranged from 0.07 to 
0.28 standard deviations above the performance of eligible non-participants (see Exhibit 18). 
Nevertheless none of these differences with the group average were significant at the .05 level. This 
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group of providers, which included the Hillsborough school district, served between 24 and 476 students 
in reading for an average of 13 to 35 hours in 2006—07. 





Exhibit 24 




Reading Achievement Gains on the FCAT of Students in Grades 1-12 by Eligibility for and 


Participation in Supplementai Educationai Services, by Provider, in Hillsborough, 

2002-03 Through 2006-07 


Provider Name 


Number of 
Participants 


Average Hours per 
Student 


Reading 


Average Across All Providers 


-0.001 


Providers Showing No Significant Gains 


Private Provider A 


41 


31 


0.28 


Private Provider S 


26 


30 


0.20 


Private Provider B 


144 


17 


0.08 


Private Provider M 


194 


17 


0.07 


Private Provider K 


161 


21 


0.04 


Private Provider N 


102 


34 


0.03 


Private Provider P 


67 


28 


0.02 


Private Provider L 


112 


16 


0.01 


Private Provider U 


49 


27 


0.002 


Private Provider Y 


24 


17 


-0.01 


Hiiisborough Schooi District 


447 


22 


-0.01 


Private Provider J 


476 


25 


-0.02 


Private Provider R 


48 


13 


-0.03 


Private Provider I 


331 


19 


-0.05 


Private Provider H 


543 


24 


-0.06 


Private Provider Q 


44 


23 


-0.07 


Private Provider V 


31 


35 


-0.07 


Private Provider 0 


68 


29 


-0.07 


Private Provider T 


30 


15 


-0.08 


Other Providers 


62 


22 


-0.001 


Totai Number of Students 


3,000 






Exhibit reads: In Hillsborough, no provider showed achievement gains for participating students that were 
significandy larger than the gains for eligible non-participants. AH these gains were also statistically 
indistinguishable from the average effect size across all supplemental educational services participants in the 
district. 


Notes: Scores reported are rank-based z-scores. The “Other Providers” category includes all providers serving less than 
20 students in mathematics. All models controlled for student eligibility for services. 


* indicates that the provider experienced gains that were statistically different from the gains of eligible non- 
participants at the .05 level. 

Shaded bars indicate that the provider experienced gains that were statistically different from the district average. 
In this exhibit, no bars are shaded, indicating that there are no statistically significant differences from the district 


average. 








Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Effect of Number of Hours of Services on Student Achievement 



In both Anchorage and Hillsborough, a majority of supplemental educational services 
participants received fewer than 40 hours of services. 

In Anchorage and Hillsborough, there was a high level of variation in the number of hours of 
supplemental educational services students received from providers. To analyze the relationship 
between hours of services and achievement gains, we developed three broad categories: a) fewer than 
20 hours of services, b) between 20 and 39 hours of services, and c) greater than 39 hours of services 
(see Exhibit 25). 







Exhibit 25 








Distribution of Supplemental Educational Services Participants by Number of Hours of 


Services Received, by Provider Type, in Anchorage and Hillsborough, 

2004-05 Through 2006-07 






All Providers 


District Provider 


Non-District Provider 




2004-05 To 2006-07 


2006-07 




Percent 


n 


Percent 


n 


Percent 


n 


Hours of Services Received 








Anchorage (Grades K-12) 








Fewer than 20 hours 


28% 


121 


33% 


45 


25% 


52 


20-39 hours 


43% 


188 


35%* 


48 


47%* 


105 


Greater than 39 hours 


27% 


129 


33%* 


45 


28%* 


8 


Total 


100% 


438 


100% 


138 


100% 


165 












Percent 


n 


Percent 


n 


Percent 


n 


Hillsborough (Grades K-12) 








Fewer than 20 hours 


36% 


3,654 


46%* 


442 


35%* 


3,212 


20-39 hours 


59% 


5,937 


45%* 


441 


60%* 


5,496 


40-59 hours 


5% 


505 


9%* 


87 


4%* 


405 


Total 


100% 


10,096 


100% 


971 


100% 


9,125 


Exhibit reads: In Anchorage, 28 percent of all participants in grades K— 12 received less than 20 hours of 
services. Among those served by the district in 2006—07, 33 percent received less than 20 hours of services 
and among those served by non-district providers, 25 percent received less than 20 hours of services during 
that year. 


Notes: * indicates a significant difference at the .05 level between the percentage of participants served by the district 
provider and non-district providers. 


Source: Anchorage School District Administrative Data, 2002—03 through 2006—07; Hillsborough School District 
Administrative Data, 2002-03 through 2006-07. 
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Between 2004—05 and 2006—07, the majority of participants in both districts received less than 40 hours 
of services. In Anchorage, less than 30 percent of participants received more than 40 hours of services, 
while in Hillsborough only 5 percent of participants received more than 40 hours of services. Jn 
Anchorage, there was not a statistically significant difference between the district provider and non- 
district providers in the percentage of their participants who received less than 20 hours of services in 
2006—07. In Hillsborough, the district provider tended to provide students slightly fewer hours of 
tutoring than other providers (see Exhibit 25). Note, however, that this exhibit combines the available 
data from 2004—05 through 2006—07, in contrast to Exhibit 10 (overall average number of hours), which 
only includes data for the 2006—07 school year. (See Appendix D for an analysis of the average number 
of hours of supplemental educational services across district and non-district providers using the data of 
2004-05 through 2006-07.) 

There was not a consistent relationship between the amount of services received 
and achievement gains in both districts. 

In Anchorage, reading achievement gains of students receiving fewer than 20 hours of tutoring from the 
district lagged 0.24 standard deviations below eligible non-participants. Students served by non-district 
providers had no statistically significant gains or losses compared to eligible non-participants, regardless 
of hours of services received (see Exhibit 26). Breaking down the analysis by hours of services, 
mathematics achievement gains were statistically indistinguishable from those of eligible non- 
participants. 



Hillsborough only provided information on hours of supplemental educational services for 2005—06 and 2006—07. 
398 students received these services in 2004—05 and it was necessary to impute the hours of supplemental educational 
services received for students during that school year. We did so by using the average hours of service observed by 
provider during 2005-06 and 2006-07. For those providers that did not serve students after 2004—05, we used the 
overall average number of hours across aU providers. 

12 Note that students served by non-district providers experienced overall larger gains in mathematics than eligible non- 
participants. But when these students are separated into different groups by hours of tutoring, the sample sEes become 
too small in Anchorage to detect statistical significance. 
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Exhibit 26 

Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 
by Provider Type and Hours of Services Received, in Anchorage, 
2002-03 Through 2006-07 




All Providers 


District Provider 


Non-District Providers 




2004-05 To 2006-07 


2006-07 




Gains 


n 


Gains 


n 


Gains 


n 


Hours of Services Received 


Mathematics 


Fewer than 20 hours 


-0.004 


70 


-0.16 


31 


0.18 


20 


20-39 hours 


-0.10 


82 


** 


18 


0.12 


37 


40 or more hours 


-0.01 


69 


** 


18 


** 


5 


Total n 




201 




67 




62 


Reading 


Fewer than 20 hours 


-0.08 


85 


-0.24* 


34 


0.01 


32 


20-39 hours 


0.07 


96 


0.01 


20 


0.08 


53 


40 or more hours 


0.05 


77 


-0.05 


29 




5 


Total n 




258 




83 




90 


Exhibit reads: In Anchorage, overall, students receiving fewer than 20 hours of supplemental educational 
services did not demonstrate greater gains in mathematics achievement than eligible non-participants. 
Specifically, students receiving 20 hours or less of supplemental educational services from the district in 
reading experienced smaller gains (-0.24 standard deviations) than eligible non-participants. 

Notes: Scores reported are rank-based 2 -scores. N is the number of student observations in treatment contributing to 
the estimate. AH models controlled for student eligibility for services. 

* indicates gains that were statistically different form the gains of eligible non-participants at the .05 level. 

** indicates that the data are not reported due to small sample si 2 e (n<20 students). 

Source; Anchorage School District Administrative Data, 2002—03 through 2006—07. 
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In HiUsborough, mathematics achievement gains were found for students who received less than 20 
hours of services and for those receiving 20 to 39 hours of services. Gains for students receiving 40 or 
more hours of services in mathematics were not statistically significant, but participant sizes for this 
group were very small (see Exhibit 27). During 2006-07, non-district providers serving students who 
received between 20 and 39 hours of services in mathematics experienced gains of about 0.1 standard 
deviations larger than those of eligible non-participants. 

Breaking down the analysis by hours of tutoring, no statistically significant differences were observable in 
reading in Hillsborough between participants and eligible non-participants. 



Exhibit 27 

Achievement Gains on the FCAT of Students in Grades 1-12 by Provider Type and 
Hours of Services Received, in Hillsborough, 2002-03 Through 2006-07 




All Providers 


District Provider 


Non-District Providers 




2004-05 To 2006-07 


2006-07 




Gains 


n 


Gains 


n 


Gains 


n 


Hours of Services Received 


Mathematics 


Fewer than 20 hours 


0.05* 


761 


0.03 


112 


0.05 


277 


20-39 hours 


0.04* 


1,096 


0.07 


104 


0.09* 


445 


40 or more hours 


0.09 


106 


0.08 


27 


0.18 


39 


Total n 




1,966 




243 




761 


Reading 


Fewer than 20 hours 


-0.01 


2,471 


-0.10 


210 


-0.03 


795 


20-39 hours 


0.002 


4,445 


0.05 


211 


0.01 


1,673 


40 or more hours 


0.03 


362 


0.13 


84 


-0.03 


84 


Total n 




7,288 




505 




2,552 


Exhibit reads: In Hillsborough, students receiving fewer than 20 hours of services in mathematics 




experienced academic gains of .05 standard deviations above eligible non-participants, a statistically 




significant difference. 
















Notes: Scores reported are rank-based 2 -scores. N is the number of student observations in treatment contributing to 


the estimate. All models controlled for student eligibility for services. 










* indicates gains that were statistically different form the gains of eligible non-participants at the .05 level. 


Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Effect of Subject of Services on Achievement 



In this study, only participants in Hillsborough that received tutoring in mathematics 
experienced larger academic gains (in mathematics) than eligible non-participants. 

Not all participants received supplemental educational services in the same subject. Because of this, it is 
important to analyze whether the subject of the service provided was related to the academic gains of 
participants. In Anchorage, no statistically significant differences in academic gains were observable 
between students who only received supplemental educational services in mathematics or reading and 
eligible non-participants (see Exhibit 28). However, those who received services in both subjects 
experienced smaller academic gains in mathematics than did eligible non-participants. 



Exhibit 28 

Achievement Gains on the Benchmark and SB A of Students in Grades 3-10 by Subject, 

in Anchorage, 2002-03 Through 2006-07 





Mathematics 


Reading 




Gains 


n 


Gains 


n 



Subject of Tutoring 



Mathematics and Reading 


-0.21* 


139 


-0.11 


137 


Mathematics Oniy 


0.04 


49 


-0.09 


50 


Reading Oniy 


0.05 


92 


0.11 


91 



Exhibit reads: In Anchorage, students who received services in both mathematics and reading 
demonstrated smaller gains in mathematics achievement, with a change in their academic achievement of .21 
standard deviations below eligible non-participants, a statistically significant difference. 

Notes: Scores reported are rank-based 2 -scores. N is the number of student observations in treatment contributing to 
the estimate. AH models controlled for student eligibility for services. 

* indicates gains that were statistically different form the gains of eligible non-participants at the .05 level. 
Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 
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In HiUsborough, participants who received supplemental educational services in reading experienced no 
larger academic gains in reading or mathematics than eligible non-participants. However, students who 
received supplemental educational services in mathematics only demonstrated a significant gain in 
mathematics achievement, 0.07 standard deviations (see Exhibit 29). Students who received 
supplemental educational services in both subjects did not experience statistically significant gains in 
either subject in comparison to eligible non-participants. 



Exhibit 29 

Achievement Gains on the FCAT of Students in Grades 1-12 by Subject, in Hillsborough, 

2002-03 Through 2006-07 





Mathematics 


Reading 




Gains 


n 


Gains 


n 



Subject of Tutoring 



Mathematics and Reading 


0.004 


608 


0.001 


608 


Mathematics Oniy 


0.07* 


1,552 


-0.02 


1,552 


Reading Oniy 


-0.01 


6,810 


0.03 


6,810 



Exhibit reads: In Hillsborough, students who received services in both mathematics and reading did not 
demonstrate greater gains in mathematics achievement than eligible non-participants. 



Notes: Scores reported are rank-based 2 -scores. N is the number of student observations in treatment contributing to 
the estimate. AH models controlled for student eligibility for services. 

* indicates gains that were statistically different form the gains of eligible non-participants at the .05 level. 
Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 



Effect of Participation by Student Characteristics 

In both districts, most student subgroups did not show significant achievement 
gains. 

In Anchorage, where sample sizes were very small, the only student subgroup showing a significant 
difference between participants and non-participants was Hispanic students, where a lower gain than 
eligible non-participants was found in mathematics (see Exhibit 30). In HiUsborough, positive 
achievement gains were found for Hispanic and limited EngUsh proficient students in mathematics. No 
significant gains were observed for African-American, white, or special education students in 
mathematics or for any subgroup in reading (see Exhibit 31). 
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Exhibit 30 

Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 
for Supplemental Educational Services Participants, 
by Student Characteristics, in Anchorage, 2002-03 Through 2006-07 






Mathematics 


Reading 




Participation Effect: 


Participation Effect: 




Gains 


n 


Gains 


n 


All Participants 


-0.05 




0.01 




Participation Among: 






African-American Students 


0.13 


49 


-0.03 


58 


Alaska Native Students 


-0.01 


42 


0.06 


54 


White Students 


-0.14 


35 


-0.17 


37 


Asian Students 


0.11 


37 


0.24 


44 


Hispanic Students 


-0.34* 


36 


-0.29 


36 


Exhibit reads: In Anchorage, African-American students who participated in supplemental 
educational services did not demonstrate greater gains in mathematics achievement than eligible 


non-participants. 










Notes: Scores reported are rank-based z-scores. N is the number of student observations in treatment 
contributing to the estimate. All models controlled for student eligibility for services. 

* indicates gains that were statistically different form the gains of eligible non-participants at the .05 level. 


Source; Anchorage School District Administrative Data, 2002—03 through 2006—07. 





Exhibit 31 

Achievement Gains on the FCAT of Students in Grades 1-12 
for Supplemental Educational Services Participants, 
by Student Characteristics, in Hillsborough, 2002-03 Through 2006-07 




Mathematics 


Reading 


Gains 


n 


Gains 


n 


All Participants 


0.05* 




-0.001 




Participation Among: 






African-American Students 


0.04 


879 


-0.01 


2,448 


Hispanic Students 


0.05* 


702 


-0.01 


3,361 


White Students 


0.08 


270 


0.05 


1,090 


Special Education Students 


0.08 


370 


-0.02 


1,901 


Limited English Preficient 


0.05* 


615 


-0.01 


3,185 


Exhibit reads: In Hillsborough, African-American students who participated in supplemental 
educational services did not demonstrate greater gains in mathematics or reading achievement than 
eligible non-participants. 

Notes: Scores reported are rank-based z-scores. N is the number of student observations in treatment 
contributing to the estimate. All models controlled for student eligibility for services. 

* indicates gains that were statistically different form the gains of eligible non-participants at the .05 level. 

Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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IV. DISTRICT COMMUNICATION WITH PARENTS 



By accepting the supplemental educational service waivers, the four districts were required to improve 
communication with parents by using multiple communications strategies, extend enrollment windows, 
and provide balanced information on aU providers. These strategies were intended to achieve one 
objective of the waivers: to help ensure that more eligible students received supplemental educational 
services. 

As evidence of meeting this requirement. Anchorage and Hillsborough submitted examples of their 
communication with parents about supplemental educational services eligibility and about their provider 
options. Anchorage submitted six documents and notes. Hillsborough submitted three documents and 
a parent handbook.^^ Two study team members independently reviewed each document looking for 
clarity (including general readability, correct use of terms, and use of the phrase “free tutoring”), bias 
towards district providers, and use of jargon. To assess readability, the Flesch-Kincaid scorers was 
calculated to determine the grade level at which each document was written. (The Flesch-Kincaid test 
measures word length and sentence length to generate a score, which is translated into a U.S. grade level 
indicator, representative of the approximate educational level to which the text is written.) Reviewers 
also determined whether the documents were available in other languages, whether districts used diverse 
outreach strategies to distribute information about supplemental educational services to parents and 
whether district materials included key details, such as contact information. 

Both Anchorage and Hillsborough used jargon and complex language in 
communication with parents. 

The parent notification documents districts distributed, such as letters, handbooks and flyers, 
demonstrated an effort to communicate necessary information about supplemental educational services 
with parents. However, the style and language used in these letters was somewhat complex. For 
example, one letter to parents in Anchorage begins with “Under the federal education law. No Child Ceft 
Behind, some students who attend Title I schools that have reached an NCLB School Improvement 
Status of Level 2 or higher are eligible for supplemental educational services.” That particular letter rated 
a 10.2 on the Flesch-Kincaid readability test, indicating the letter was written just above the tenth grade 
level. 

Similarly, the Hillsborough supplemental educational services parent handbook included questions under 
the heading “General Information.” In response to the question “What are supplemental educational 
services?” the parent handbook explained, “According to the No Child Ceft Behind Act, supplemental 
educational services are free tutoring and other supplemental academic enrichment services that are in 
addition to instruction provided during the school day.” The Flesch-Kincaid readability rating for this 
document was 13.2, which suggests it would be challenging for parents who had not completed high 
school. 



Guidance to districts that received waivers from the U.S. Department of Education included a checklist of documents 
and data that districts were encouraged to submit as evidence of meeting the waiver requirements. Note that the 
materials examined were for the 2006—07 school year. 

See Flesch, R. (1948); A new readability yardstick, of Applied Vsjcholog), Vol. 32, pp. 221-233; and Kincaid,}. P.; 
Fishburne, R. P., Jr.; Rogers, R. L.; and Chissom, B. S. (1975); Derivation of new readability formulas (Automated Readability 
Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel, Research Branch Report 8-75, Millington, TN: 
Naval Technical Training, U. S. Naval Air Station, Memphis, TN 
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Both Anchorage and Hillsborough used multiple communication strategies to reach 
eligible families. 



Districts receiving the waiver were required to use multiple communication strategies to inform eligible 
families about supplemental educational services in order to increase participation. Anchorage indicated 
that it used multiple methods of communication. For example, it mailed notification letters to all 
households in which a student was eligible for supplemental educational services on August 3 and 
September 7. These letters covered the supplemental educational services options and specifics and 
advertised upcoming informational meetings. Anchorage sent these letters in English, Hmong, Samoan, 
Spanish, and Tagalog. On August 4, 2006, the district published AYP results and supplemental 
educational services information in the Anchorage Daily News newspaper. Between September and 
November Anchorage posted one hundred posters in ten schools advertising informational meetings, 
and each school put notices in their school newsletters during September and October. Also during 
September and October, the district held informational meetings in each of the ten schools in which 
students were eligible for supplemental educational services, and on September 1 9, sent automated 
phone calls to households with eligible students. On September 20, Anchorage held a provider fair at 
one middle school. In November, schools required to offer supplemental educational services set up 
tables at their entrance on parent- teacher conference nights. In addition, information was continually 
available on the Anchorage school district website. Finally, in April and May of 2007, the district began 
working with the state’s Parent Information Resource Center to form partnerships with faith-based and 
community-based organizations to provide information to parents about supplemental educational 
services. 

Hillsborough also used several methods to reach families. It sent families with eligible students a 
Supplemental Educational Services Parent Handbook for the 2006—07 school year. The handbook 
contained a welcome letter, general information questions, a provider “at-a-glance” chart, a description 
of delivery models (in-home, distance learning, off-site, on-site), parent satisfaction survey results from 
the 2005—06 school year, a list of acronyms and phrases, detailed profiles of each provider, and a 
provider fair announcement. The district also offered ten provider fairs in August. All fairs were held 
on weeknights from 5:30pm to 7:30pm. The announcement indicated that door prizes and refreshments 
would be available. A brochure about the district provider, Hillsborough County Public Schools 
Academy for Success (HCPSAFS), was available in both English and Spanish. Parents who chose 
HCPSAFS received a thank you letter dated October 5. The letter informed them that HCPSAFS was 
stiU hiring staff and would contact them within four to five weeks about scheduling a goal-setting 
meeting. In the letter, HCPSAFS gave parents the option to chose another provider if they wished. 

This letter was sent in both English and Spanish. 

Both Anchorage and Hillsborough provided extended enrollment windows for 
families to sign up for services. 

Another stipulation of the waiver was that districts would offer extended or additional enrollment 
windows to give more families the opportunity to sign up and participate in supplemental educational 
services. In Anchorage, notification letters were sent in August and September. Enrollment began 
September 7 with the application due date of October 20. The district offered an additional window 
beginning with the parent- teacher conferences November 8 and ending with an application due date of 
November 22. The district indicated that applications that arrived after the due date were approved. 

According to Hillsborough’s communication with the U.S. Department of Education, its enrollment 
began during pre-planning days in July with a steady recruitment process going through the first soft 
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deadline date of October 13 — a deadline required by state law. Enrollment then continued until January 
26 for any interested parent of an eligible student. 

Both Anchorage and Hillsborough provided balanced information on all providers 
and showed no bias towards their own programs. 

Districts receiving the waiver needed to demonstrate that they provided eligible families with balanced 
information on all providers. Anchorage submitted a one-page flyer that lists thirteen providers in 
alphabetical order. Contact information for each provider was listed. Anchorage School District was 
sixth on the list and no bias was evident. 

In the Hillsborough Supplemental Educational Services Parent Handbook, each provider was listed in 
alphabetical order. The district’s Academy for Success was second in that order, but there was no 
indication it was the district provider aside from the contact email address. 
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V. CONCLUSIONS 



This examination of the implementation of supplemental educational services shows that in Anchorage 
and Hillsborough the percentage of eligible students participating in supplemental educational services 
increased over time, reaching 11 and 14 percent, respectively, in 2006—07. 

In both Anchorage and Hillsborough, students who were eligible for supplemental educational services 
had significantly lower levels of prior achievement in mathematics and reading than students who were 
not eligible. In both districts students who participated in supplemental educational services in 2006—07 
had lower levels of prior academic achievement in 2005—06 than eligible students who chose not to 
participate. 

In both districts, students served by non-district providers showed larger academic achievement gains 
than eligible non-participants in mathematics (but not in reading). Students served by the two district 
providers (Anchorage and Hillsborough) did not show gains in either subject compared with eligible 
non-participants. Looking at individual non-district providers, only one such provider in Hillsborough 
showed an achievement gain (in mathematics) relative to eligible non-participants. No individual 
provider in Anchorage showed any significant gains relative to non-participants. 

Overall, in Anchorage, participants receiving supplemental education services from any provider did not 
show a statistically significant difference in academic achievement gains compared with eligible 
non-participants. In Hillsborough, the overall results across all providers showed achievement gains in 
mathematics but not in reading. 

While there is room for improvement regarding district communication with parents about supplemental 
educational services, both Anchorage and Hillsborough made efforts to inform eligible families about 
the program offerings and to help ensure that more eligible students receive supplemental educational 
services. To this end, both districts used multiple communication methods and provided unbiased 
program information. 
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APPENDIX A: 

DESCRIPTION OF DISTRICT DATA 



Anchorage 

The Anchorage school district provided student-level data for the school years 2002—03 through 
2006—07. Demographic and assessment data were provided for aU years and aU students in the district, 
while detailed information on supplemental educational services was provided for 2004—05 through 
2006—07. In these latter years, data were provided on the subjects in which students were tutored, the 
total number of hours of tutoring they received, and the provider they chose. Student eligibility and 
participation in supplemental educational services was provided for all three years. 

Hillsborough 

The Hillsborough school district provided student-level data for all students taking Florida’s 
Comprehensive Assessment Test (FCAT) in 2002—03 through 2006—07, which included students in 
grades 3—10 in 2005—06 and 2006—07. The district also provided information on student eligibility, 
participation, provider used, and number of hours of tutoring received. These comprehensive databases 
included all students in the district during the whole period of analysis, 2002—03 through 2006—07. 
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APPENDIX B: 

SUPPLEMENTAL EXHIBITS 



Exhibit B.1 

Supplementai Service Participation Rates in Grades K-12, 
by Race-Ethnicity, in Anchorage, 2006-07 




Eligible 

Non-Participants 


Participants 


Participation 

Rate 


% African-American 


334 


64 


16.1% 


% Alaska Native 


654 


61 


8.5% 


% White 


417 


51 


10.9% 


% Asian 


678 


48 


6.6% 


% Multi-Racial 


271 


44 


14.0% 


% Hispanic 


197 


37 


1 5.8% 


% American Indian 


31 


3 


8.8% 


Total 


2,582 


308 


10.7% 


Exhibit reads: In 2006—07, 334 African-American students were eligible for 


supplemental educational services but did not participate in the program in Anchorage. 


The number of African-American participants was equal to 64 during that year, or 


16.1 percent of all eligible African-American students. 




Source: Anchorage School District Administrative Data, 2002—03 through 2006- 


-07. 





Exhibit B.2 






Supplemental Service Participation Rates in Grades K-12, 


by Race-Ethnicity, in Hillsborough, 2006-07 






Eligible 

Non-Participants 


Participants 


Participation 

Rate 


% African-American 


13,113 


2,034 


13.4% 


% White 


6,372 


838 


1 1 .6% 


% Asian 


510 


70 


12.1% 


% Multi-Racial 


1,597 


255 


13.8% 


% Hispanic 


15,441 


2,966 


16.1% 


% American Indian 


71 


9 


1 1 .3% 


Total 


37,104 


6,172 


14.3% 


Exhibit reads: In 2006—07, 13,113 African-American students were eligible for 
supplemental educational services but did not participate in the program in Hillsborough. 
The number of African-American participants was equal to 2,034 during that year, or 


13.4 percent of all eligible African-American students. 




Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit B.3 

Number and Percent of Supplemental Educational Services Participants in Grades K-12 

by Subject, in Anchorage, 2006-07 




Exhibit reads: In Anchorage in 2006—07, 135 students participated in supplemental educational services in 
reading only, representing 44 percent of all participants. 

Source: Anchorage School District Administrative Data, 2006—07. 
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Exhibit B.4 

Number and Percent of Supplemental Educational Services Participants in Grades K-12 

by Subject, in Hillsborough, 2006-07 




Exhibit reads: In Hillsborough, 4,857 students received supplemental educational services in reading only in 
2006-07, representing 79 percent of all participants. 

Source; Hillsborough School District Administrative Data, 2006—07. 
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Exhibit B.5 

Subject of Supplemental Services in Grades K-12, by Provider, in Anchorage, 2006-07 




All Participants 


Anchorage School 
District 


Other Providers 




Number of 


Percent of 


Number of 


Percent of 


Number of 


Percent of 




Students 


Total 


Students 


Total 


Students 


Total 


Only Reading 


135 


43.8% 


52 


37.7% 


83 


48.8% 


Only Math 


51 


16.6% 


11 


8.0% 


40 


23.5% 


Reading and Math 


72 


23.4% 


54 


39.1% 


18 


10.6% 


No Information 


50 


1 6.2% 


21 


15.2% 


29 


17.1% 


Total 


308 


100% 


138 


100% 


170 


100% 


Exhibit reads: In 2006—07, 135 students received supplemental educational services only in reading in 


Anchorage. This represented 43.8 percent of all students receiving supplemental educational services in any 


subject during that year. 












Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 







Exhibit B.6 

Subject of Supplemental Services in Grades K-12, by Provider, in Hillsborough, 2006-07 




All Participants 


Hillsborough School 
District 


Other Providers 




Number of 


Percent of 


Number of 


Percent of 


Number of 


Percent of 




Students 


Total 


Students 


Total 


Students 


Total 


Only Reading 


4,857 


78.7% 


668 


68.8% 


4,204 


80.4% 


Only Math 


1,283 


20.8% 


296 


30.5% 


999 


19.1% 


Reading and Math 


32 


0.5% 


7 


0.7% 


27 


0.5% 


Total 


6,172 


100% 


971 


100% 


5,230 


100% 


Exhibit reads: In 2006—07, 4,857 students received su 


pplemental educational services only in reading in 


Hillsborough. This represented 78.7 percent of all students receiving supplemental educational services in 


any subject during that year. 












Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit B.7 

Average Number of Hours of Supplemental Educational Services Received by 
Participants in Grades K-12, by Provider, in Anchorage and Hillsborough 



31 * 32 * 32 




Anchorage Hlstxiinigh 

(200+ 05 trough (200+ csmnou^ 
ZOOfr-OT) 2006-07) 



H Oistrlct PratMer |_J OOKrPnnklere |_| Total 



Exhibit reads: In Anchorage, students who received supplemental educational services from the district 
provider received, on average, 31 hours of tutoring. 

* indicates differences in the hours of services provided by district and non-district providers that were 
statistically different at the .05 level. 

Source: Anchorage School District Administrative Data, 2004—05 through 2006—07, and Hillsborough School District 
Administrative Data, 2004—05 through 2006—07. 
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APPENDIX C: ROBUST STANDARD ERRORS 
OF FIXED-EFFECTS REGRESSIONS 



Exhibit C.1 

Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 
by Participation in Supplemental Educational Services, in Anchorage, 

2002-03 Through 2006-07 



Mathematics 



Reading 





Gains 


n 


Gains 


n 


Participation 


Overaii Effect 


(0.075) 


221 


(0.052) 


258 


First Year Effect 


(0.076) 


215 


(0.052) 


255 


Source; Anchorage School District Administrative Data, 2002—03 through 2006—07. 






Exhibit C.2 






Achievement Gains on the FCAT of Students in Grades 1-12 by Participation 


in Supplemental Educational Services, in Hillsborough, 






2002-03 Through 2006-07 








Mathematics 


Reading 




Gains 


n 


Gains 


n 


Participation 


Overaii Effect 


(0.014) 


1,966 


(0.009) 


7,288 


Number of Years of Participation 


First Year Effect 


(0.014) 


1,683 


(0.009) 


6,092 


Muitipie Year Effect 


(0.043) 


283 


(0.019) 


1,196 


Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit C.3 






Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 


by Provider Type, in Anchorage, 2002-03 Through 2006-07 






Mathematics 


Reading 




Gains 


n 


Gains 


n 


Provider Type 






District Provider 


(0.097) 


67 


(0.109) 


83 


Non-District Providers 


(0.087) 


62 


(0.075) 


90 


Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 



Exhibit C.4 

Mathematics Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 
by Eligibility for and Participation in Supplemental Educational Services, by Provider, in 

Anchorage, 2002-03 Through 2006-07 


Provider Name 


Number of Participants 


Average Hours per 
Student 


Mathematics 


Average Across All Providers 


(0.097) 


Providers Showing No Significant Effects 


Private Provider A 


46 


31 


(0.109) 


Anchorage School District 


67 


29 


(0.097) 


Other Providers 


16 


28 


(0.172) 


Total Number of Students 


129 






Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit C.5 

Reading Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 by 
Eligibility for and Participation in Supplemental Educational Services, by Provider, in 

Anchorage, 2002-03 Through 2006-07 


Provider Name 


Number of Participants 


Average Hours per 
Student 


Reading 


Average Across All Providers 


(0.109) 


Providers Showing No Significant Effects 


Private Provider A 


70 


27 


(0.101) 


Anchorage Schooi District 


83 


33 


(0.109) 


Other Providers 


20 


26 


(0.178) 


Total Number of Students 


173 






Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 





Exhibit C.6 






Achievement Gains on the FCAT of Students in Grades 1-12 
by Provider Type, in Hillsborough, 2002-03 Through 2006-07 






Mathematics 


Reading 




Gains 


n 


Gains 


n 


District Provider 


(0.037) 


234 


(0.046) 


447 


Non-District Providers 


(0.025) 


743 


(0.019) 


2,503 


District and Non-District Providers® 


(0.119) 


18 


(0.177) 


50 


Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit C.7 

Mathematics Achievement Gains on the FCAT of Students in Grades 1-12 by Eligibility for 
and Participation in Supplemental Educational Services, by Provider, in Hillsborough, 

2002-03 Through 2006-07 


Provider Name 


Number of 
Participants 


Average Hours per 
Student 


Mathematics 


Average Across All Providers 


(0.037) 


Providers Showing Significant Gains 


Private Provider S 


20 


27 


(0.143) 


Providers Showing No Significant Gains 


Private Provider W 


22 


9 


(0.177) 


Private Provider Q 


49 


24 


(0.087) 


Private Provider N 


40 


35 


(0.044) 


Private Provider I 


158 


18 


(0.054) 


Private Provider H 


213 


24 


(0.028) 


Hillsborough School District 


234 


21 


(0.037) 


Private Provider L 


55 


16 


(0.076) 


Private Provider K 


52 


19 


(0.076) 


Private Provider B 


44 


16 


(0.111) 


Other Providers 


128 


24 


(0.080) 


Totai Number of Students 


995 






Source: Hillsborough School District Administrative Data, 2002 


-03 through 2006-07. 
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Exhibit C.8 

Reading Achievement Gains on the FCAT of Students in Grades 1-12 by Eligibility for and 
Participation in Supplemental Educational Services, by Provider, in Hillsborough, 

2002-03 Through 2006-07 


Provider Name 


Number of 
Participants 


Average Hours per Student 


Reading 


Average Across All Providers 


(0.046) 


Providers Showing No Significant Gains 


Private Provider A 


41 


31 


(0.185) 


Private Provider S 


26 


30 


(0.218) 


Private Provider B 


144 


17 


(0.080) 


Private Provider M 


194 


17 


(0.056) 


Private Provider K 


161 


21 


(0.082) 


Private Provider N 


102 


34 


(0.069) 


Private Provider P 


67 


28 


(0.068) 


Private Provider L 


112 


16 


(0.058) 


Private Provider U 


49 


27 


(0.028) 


Private Provider Y 


24 


17 


(0.048) 


Hiiisborough Schooi District 


447 


22 


(0.047) 


Private Provider J 


476 


25 


(0.038) 


Private Provider R 


48 


13 


(0.064) 


Private Provider I 


331 


19 


(0.043) 


Private Provider H 


543 


24 


(0.039) 


Private Provider Q 


44 


23 


(0.094) 


Private Provider V 


31 


35 


(0.087) 


Private Provider 0 


68 


29 


(0.123) 


Private Provider T 


30 


15 


(0.145) 


Other Providers 


62 


22 


(0.126) 


Totai Number of Students 


3,000 






Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit C.9 

Achievement Gains on the Benchmark and SBA of Students in Grades 3- 
by Provider Type and Hours of Services Received, in Anchorage, 
2002-03 Through 2006-07 


-10 




All Providers 


District Provider 


Non-District Providers 




Gains 


n 


Gains 


n 


Gains 


n 


Hours of Services Received 


Mathematics 


Less than 20 hours 


(0.078) 


70 


(0.081) 


31 


(0.124) 


20 


20-39 hours 


(0.140) 


82 


(0.326) 


18 


(0.091) 


37 


40 or more hours 


(0.167) 


69 


(0.071) 


18 


(0.216) 


5 


Total n 




201 




67 




62 


Reading 


Less than 20 hours 


(0.075) 


85 


(0.080) 


34 


(0.068) 


32 


20-39 hours 


(0.089) 


96 


(0.149) 


20 


(0.129) 


53 


40 or more hours 


(0.109) 


77 


(0.240) 


29 


(0.247) 


5 


Total n 




258 




83 




90 


Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 







Exhibit C.10 








Achievement Gains on the FCAT of Students in Grades 1-12 by Provider Type and 


Hours of Services Received, in Hillsborough, 2002-03 Through 2006-07 




All Providers 


District Provider 


Non-District Providers 




Gains 


n 


Gains 


n 


Gains 


n 


Hours of Services Received 


Mathematics 


Less than 20 hours 


(0.037) 


761 


(0.060) 


112 


(0.045) 


277 


20-39 hours 


(0.024) 


1,096 


(0.050) 


104 


(0.027) 


445 


40 or more hours 


(0.093) 


106 


(0.071) 


27 


(0.146) 


39 


Total n 




1,966 




243 




761 


Reading 


Less than 20 hours 


(0.028) 


2,471 


(0.069) 


210 


(0.028) 


795 


20-39 hours 


(0.022) 


4,445 


(0.050) 


211 


(0.023) 


1,673 


40 or more hours 


(0.069) 


362 


(0.184) 


84 


(0.046) 


84 


Total n 




7,288 




505 




2,552 


Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit C.11 






Achievement Gains on the Benchmark and SB A of Students in Grades 3-10 by Subject, 


in Anchorage, 2002-03 Through 2006-07 






Mathematics 


Reading 




Gains 


n 


Gains 


n 


Subject of Tutoring 


Mathematics and Reading 


(0.069) 


139 


(0.065) 


137 


Mathematics Oniy 


(0.100) 


49 


(0.079) 


50 


Reading Oniy 


(0.095) 


92 


(0.093) 


91 


Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 





Exhibit C.12 






Achievement Gains on the FCAT of Students in Grades 1-12 by Subject, in Hillsborough, 




2002-03 Through 2006-07 








Mathematics 


Reading 




Gains 


n 


Gains 


n 


Subject of Tutoring 


Mathematics and Reading 


(0.018) 


608 


(0.019) 


608 


Mathematics Oniy 


(0.017) 


1,552 


(0.016) 


1,552 


Reading Oniy 


(0.010) 


6,810 


(0.010) 


6,810 


Source; Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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Exhibit C.13 






Achievement Gains on the Benchmark and SBA of Students in Grades 3-10 


for Supplemental Educational Services Participants, 




by Student Characteristics, in Anchorage, 2002-03 Through 2006-07 


Anchorage 




Mathematics 


Reading 




Participation Effect: 


Participation Effect: 




Gains 


n 


Gains 


n 


All Participants 


(0.075) 


221 


(0.052) 


258 


Participation Among: 






African-American Students 


(0.163) 


49 


(0.087) 


58 


Aiaska Native Students 


(0.136) 


42 


(0.091) 


54 


White Students 


(0.190) 


35 


(0.114) 


37 


Asian Students 


(0.174) 


37 


(0.130) 


44 


Hispanic Students 


(0.131) 


36 


(0.203) 


36 


Source: Anchorage School District Administrative Data, 2002—03 through 2006—07. 



Exhibit C.14 

Achievement Gains on the FCAT of Students in Grades 1-12 
for Supplemental Educational Services Participants, 
by Student Characteristics, in Hillsborough, 2002-03 Through 2006-07 




Mathematics 


Reading 


Gains 


n 


Gains 


n 


All Participants 


(0.014) 


1,966 


(0.009) 


7,288 


Participation Among: 






African-American Students 


(0.019) 


879 


(0.015) 


2,448 


Hispanic Students 


(0.021) 


702 


(0.013) 


3,361 


White Students 


(0.076) 


270 


(0.025) 


1,090 


Speciai Education Students 


(0.051) 


370 


(0.020) 


1,901 


Limited Engiish Proficient 


(0.021) 


615 


(0.013) 


3,185 


Source: Hillsborough School District Administrative Data, 2002—03 through 2006—07. 
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